Elamur Research⌕
☰
home / papers / 2026 / Say Again? The Limits of Whisper with Conversation. A Case Study on the KIParla Corpus.

Say Again? The Limits of Whisper with Conversation. A Case Study on the KIParla Corpus.

Paper · 2026 · Martina Simonotti, Ludovica Pannitto, Caterina Mauri et al.

Summary

The paper asks how well Whisper preserves interactional material in spontaneous Italian dialogue, where ASR tools often discard short items as noise. The authors focus on backchannels, self- and other-initiated repairs, and filled pauses. They used 26 two-speaker conversations from the KIParla corpus (interviews, student–professor meetings and free talk), and an expert linguist manually annotated these phenomena in INCEpTION. Audio went through the DIT.DaT pipeline, which uses PyAnnote for diarization and Whisper large-v3 for transcription, under four segmentation settings (A–D). The model was kept fixed while Optuna tuned decoding parameters under two objectives: plain WER, and an Interaction-aware variant that rewards keeping target conversational tokens. Results were assessed with global WER and a Mean Match Ratio over annotated events. Neither objective clearly outperformed the other overall. Recognition depended strongly on the phenomenon: Whisper tends to flatten repairs into a single fluent version and often omits brief conversational tokens. This suggests a structural limitation rather than a tuning issue, and it warns Conversation Analysis researchers against relying on WER alone.

Key points

  • Data: 26 dialogues from KIParla totalling 15h48m.
    • The first 20 minutes of each were annotated, giving 8h40m tagged with BC, SR, OR and FP labels.
    • The remainder was split into Subset A (3h25m) for decoding optimization and Subset B (3h22m) for a control check.
  • Interaction-aware objective: it adds deliberately small weights (λ = 0.3, µ = 0.1) that reward correctly produced target tokens and penalize suppressed ones. Target tokens come from a fixed list of forms (e.g., eh, ehm, mh, sì, okay, cioè).
  • Control results on Subset B: WER was 34.23% for the WER objective and 33.24% for the Interaction-aware one. In both cases, deletions made up about 58% of the alignment outcomes.
  • Frequent substitutions: the most common was filled pause "eh" transcribed as "e" (about 50 cases). Other typical cases were "mh" or "sì" rendered as "grazie", which points to normalization of conversational tokens.
  • Availability: data and code are released at github.com/KIParla/say-again, and the DIT.DaT pipeline is also public on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BolognaBologna · Italy17 works · 1,821 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Martina Simonotti, Ludovica Pannitto, Caterina Mauri, Adriano Ferraresi and Gabriele Carioli. 2026. Say Again? The Limits of Whisper with Conversation. A Case Study on the KIParla Corpus. Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2so5y449gb4w

@inproceedings{simonotti2026say,
  title = {{Say Again? The Limits of Whisper with Conversation. A Case Study on the KIParla Corpus.}},
  author = {Martina Simonotti, Ludovica Pannitto, Caterina Mauri, Adriano Ferraresi and Gabriele Carioli},
  booktitle = {Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2so5y449gb4w},
  url = {https://doi.org/10.63317/2so5y449gb4w}
}

References

40 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paul Drew. 1997. ‘open’ class repair initiators in response to sequential sources of troubles in conversation. Journal of Pragmatics, 28(1):69– 101. find
  2. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework.
  3. Giolo Fele. 2007. L’analisi della conversazione. Il Mulino, Bologna. find
  4. Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whisperx: Time-accurate speech transcription of long-form audio. find
  5. Jan Gorisch and Thomas Schmidt. 2024. Evaluating workflows for creating orthographic transcripts for oral corpora by transcribing from scratch or correcting ASR-output. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COL… find
  6. Peter Blomsma, Julija Vaitonyte, Gabriel Skantze, and Marc Swerts. 2024. Backchannel behavior is idiosyncratic. Language and Cognition, 16:1–24. find
  7. Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. 2019. pyannote.audio: neural building blocks for speaker diarization. find
  8. Gail Jefferson. 2004. Glossary of transcript symbols with an introduction. In Gene H. Lerner, editor, Conversation Analysis: Studies from the First Generation, chapter 2, page 13–31. John Benjamins, Amsterdam / Philadelphia. find
  9. N. Christenfeld, S. Schachter, and F. Bilous. 1991. Filled pauses and gestures: It’s not coincidence. Journal of Psycholinguistic Research, 20:1–10. find
  10. Dietrich Klakow and Jochen Peters. 2002. Testing the correlation of word error rate and perplexity. Speech Communication, 38(1):19–28. find
  11. Eve V. Clark. 2020. Conversational repair and the acquisition of language. Discourse Processes, 57(5-6):441–459. find
  12. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The inception platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of System Demonstrations of the 27th International Conference on Computational Linguistics,…
  13. Francisco Cossavella and Jazmín Cevasco. 2021. The importance of studying the role of filled pauses in the construction of a coherent representation of spontaneous spoken discourse. Journal of Cognitive Psychology, 33(2):172–186. find
  14. Ronald Cumbal, Birger Moell, José Lopes, and Olov Engwall. 2021. “you don’t understand me!”: Comparing asr results for l1 and l2 speakers of swedish. pages 4463–4467. find
  15. Andreas Liesenfeld, Alianda Lopez, and Mark Dingemanse. 2023. The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems. find
  16. Christina Dideriksen, Riccardo Fusaroli, Kristian Tylén, Mark Dingemanse, and Morten H. Christiansen. 2019. Contextualizing conversational strategies: Backchannel, repair and linguistic alignment in spontaneous and task-oriented conversations. In Annual Meeting of the Cognitive Science Society. find
  17. Alianda Lopez, Andreas Liesenfeld, and Mark Dingemanse. 2022. Evaluation of automatic speech recognition for conversational speech in Dutch, English and German: What goes missing? In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), pages 135–143, Potsdam, Germany. find
  18. KONVENS 2022 Organizers. find
  19. Mark Dingemanse and N. J. Enfield. 2015. Other-initiated repair across languages: towards a typology of conversational structures. Open Linguistics, 1(1). find
  20. Caterina Mauri, Silvia Ballarè, Eugenio Goria, Massimo Cerruti, and Francesco Suriano. 2019. find
  21. KIParla corpus: A new resource for spoken Italian. In Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019), pages 243–249, Bari, Italy. CEUR Workshop Proceedings. find
  22. Hiroyoshi Yamasaki, Jérôme Louradour, Julie Hunter, and Laurent Prévot. 2023. Transcribing and aligning conversational speech: A hybrid pipeline applied to french conversations. find
  23. Vicky Zayats, Trang Tran, Richard Wright, Courtney Mansfield, and Mari Ostendorf. 2019. Disfluencies and human speech transcription errors. In Interspeech 2019, pages 3088–3092. find
  24. Max Planck Institute for Psycholinguistics. 2025. ELAN (Version 7.0). The Language Archive, Nijmegen. Computer software. find
  25. Daniela Mereu, Francesco Cangemi, and Martine Grice. 2024. Backchannels are not always very short utterances. the case of italian multi-unit backchannels. Journal of Pragmatics, 228:1–16. 10. find
  26. Pilar Pernas and Margarita Borreguero Zuloaga. 2010. Cortesia e scortesia in un contesto di apprendimento linguistico: la gestione dei turni. In Marcello Pettorino, Antonietta Giannini, and Francescamaria Dovetto, editors, La Comunicazione Parlata 3. Atti Del Congresso Internazionale (Napoli, 23-25… find
  27. Language Resource References Bernasconi, Beatrice and Goria, Eugenio. 2026. StraParlaTO. Università degli Studi di Torino, 0.1.0. find
  28. Cerruti, Massimo and Ballarè, Silvia. 2020. Modulo ParlaTO. find
  29. Mauri, Caterina and Ballarè, Silvia and Zucchini, Eleonora. 2024a. KIPasti. distributed via ILC-CNR for CLARIN-IT repository hosted at Institute for Computational Linguistics “A. Zampolli”, 1.1.0. PID http://hdl.handle.net/20.500.11752/OPEN-2124. ↗
  30. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  31. Emanuel Schegloff, Gail Jefferson, and Harvey Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Language, 53:361–382. find
  32. Mauri, Caterina and Ballarè, Silvia and Zucchini, Eleonora. 2024b. ParlaBO. distributed via ILC-CNR for CLARIN-IT repository hosted at Institute for Computational Linguistics “A. Zampolli”, 1.1.0. PID http://hdl.handle.net/20.500.11752/OPEN-2126. ↗
  33. Loredana Schettino and Violetta Cataldo. 2019. Lexicalized pauses in italian. pages 189–192. find
  34. Mauri, Caterina and Goria, Eugenio and Ballarè, Silvia. 2019. Modulo KIP. find
  35. Lorenzo Spreafico. 2012. Le pause piene nel parlato plurilingue. In Lessico e lessicologia: atti del XLIV Congresso internazionale di studi della Società di linguistica italiana (SLI), number 56 in Pubblicazioni della Società di linguistica italiana, pages 355–368, Roma. Bulzoni. find
  36. Zucchini, Eleonora and Ballarè, Silvia and Mauri, Caterina. 2026. StraParlaBO. Alma Mater Studiorum - Università di Bologna, 0.1.0. find
  37. Muhammad Umair, Julia Mertens, Saul Albert, and J. Ruiter. 2022. Gailbot: An automatic transcription system for conversation analysis. Dialogue & Discourse, 13:63–95. find
  38. V.N. Vitale, L. Schettino, and F. Cutugno. 2024. Rich speech signal: exploring and exploiting end-to-end automatic speech recognizers’ ability to model hesitation phenomena. In Proc. find
  39. Interspeech 2024, pages 222–226. find
  40. Nigel Ward and Wataru Tsukahara. 2000. Tsukahara, w.: Prosodic features which cue backchannel responses in english and japanese. journal of pragmatics 23, 1177-1207. Journal of Pragmatics, 32:1177–1207. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/simonotti-2026-say-again-limits-whisper-conversation/