Elamur Research⌕
☰
home / papers / 1999 / Speech repains, intonational phrases, and discourse markers: modeling speakers' utterances in spoken dialogue

Speech repains, intonational phrases, and discourse markers: modeling speakers' utterances in spoken dialogue

Paper · 1999 · Peter A. Heeman and James F. Allen

Summary

Heeman and Allen address a basic problem for spoken dialogue systems: working out what the speaker actually meant to say. A turn has to be split into utterance-like units, and the words the speaker abandoned have to be removed. These include repeated or replaced stretches of speech and editing terms such as "um" or "I mean". The authors argue that three tasks are tightly coupled: finding intonational phrase boundaries, resolving speech repairs, and recognizing discourse markers like "so" or "well". All three also interact with POS tagging and word prediction. Their solution is a single statistical language model in which the speech recognition objective is redefined. The model jointly hypothesizes words, POS tags, discourse-marker usage, repair structure and phrase boundaries, using silence as its only acoustic evidence. On the hand-transcribed Trains corpus, solving the tasks jointly beats handling them separately. Modelling repairs and phrasing also improves POS tagging and perplexity. This matters because spontaneous speech is full of disfluencies that conventional language models and taggers handle poorly.

Key points

  • Data is the Trains corpus of task-oriented human-human dialogues. 23% of turns contain at least one repair, rising to 54% for turns of 10+ words, and reparanda plus editing terms make up about 10% of all words.
  • Repairs are split into three types: fresh starts, modification repairs and abridged repairs. Each repair is analysed as reparandum, interruption point, optional editing term and alteration.
  • Overall results on transcripts:
    • turn-internal intonational boundaries: 71.8% recall, 70.8% precision;
    • discourse markers: 97.3% recall, 96.3% precision;
    • repairs detected and corrected: 65.9% recall, 74.3% precision.
  • Ablations show the gains from joint modelling:
    • with phrase modelling added, detecting repairs gets 7.4% better and correcting them 6.3% better;
    • repair and phrase modelling together improve discourse-marker identification by 15.4%;
    • modelling repairs and phrases improves POS tagging by 9.5% and perplexity by 7.0%.
  • The discourse-marker error rate is 6.4%, against 37.3% for Litman (1996), though the authors note the comparison is not direct.
  • Rescoring word graphs with the POS-based model gave about a one-point absolute WER reduction. Early Japanese experiments suggest the model is not tied to one language.

Interesting ideas

  • I1. Treat discourse-marker use as a distinct tag on a word like "so" or "well", so the tagger itself decides between the sentential reading and the marker reading.
  • I2. Correct a repair by identifying the reparandum and editing term, but keep the removed material available, since it can still carry information such as the antecedent of a pronoun.
  • I3. Abridged repairs show that word overlap between neighbouring stretches (e.g., "need to" vs "manage to") can be spurious, so a repetition-based heuristic needs positional context such as phrase boundaries.
  • I4. The evaluation removes one jointly modelled component at a time and measures how every other task degrades, which reveals how the tasks depend on each other.
  • I5. Segmentation into utterances, repair resolution and tagging are scored as one joint best hypothesis instead of a chain of hard decisions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en tagger could give marker readings their own tags, so a parser never forces "well" or "so" into an argument slot.
  • I2 (frames): Coreference in prag could search a dropped reparandum for antecedents while frames are built only from the cleaned utterance.
  • I3 (tree): A disfluency pre-pass before UD parsing should use sentence-position cues rather than surface matching alone, avoiding deletion of legitimate parallel structure.
  • I4 (learn): Mova's evaluation could ablate each pipeline layer and report the effect on all downstream layers, exposing which explanations other steps actually rely on.
  • I5 (tree): The parser could keep a few segmentation or disfluency hypotheses and let parse scores choose, rather than committing early in tokenization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksDiscourse analysis · 466 works

Cite

Cite as Peter A. Heeman and James F. Allen. 1999. Speech repains, intonational phrases, and discourse markers: modeling speakers' utterances in spoken dialogue. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@misc{heeman1999speech,
  title = {{Speech repains, intonational phrases, and discourse markers: modeling speakers' utterances in spoken dialogue}},
  author = {Peter A. Heeman and James F. Allen},
  year = {1999},
  url = {https://aclanthology.org/J99-4003/}
}

References

58 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, James F., Lenhart K. Schubert, George Ferguson, Peter Heeman, Chung Hee Hwang, Tsuneaki Kato, Marc Light, Nathaniel Martin, Bradford Miller, Massimo Poesio, and David Traum. 1995. The Trains project: A case study in building a conversational planning agent. Journal of Experimental and Theoretical AI, 7:7-48. ↗
  2. Bahl, Lalit R., J. K. Baker, Frederick Jelinek, and Robert L. Mercer. 1977. Perplexity — A measure of the difficulty of speech recognition tasks. In Proceedings of the 94th Meeting of the Acoustical Society of America. find
  3. Bahl, Lalit R., Peter E Brown, Peter V. deSouza, and Robert L. Mercer. 1989. A tree-based statistical language model for natural language speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 36(7):1001-1008. find
  4. Bard, Ellen G. and Robin J. Lickley. 1997. On not remembering disfluencies. In Proceedings of the 5th European Conference on Speech Communication and Technology, pages 2855-2858. find
  5. Beach, Cheryl M. 1991. The interpretation of prosodic patterns at points of syntactic structure ambiguity: Evidence for cue trading relations. Journal of Memory and Language, 30(6):644-663. ↗
  6. Bear, John, John Dowding, and Elizabeth E. Shriberg. 1992. Integrating multiple knowledge sources for detection and correction of repairs in human-computer dialogue. In Proceedings of the 30th Annual Meeting, pages 56-63. Association for Computational Linguistics. find
  7. Bear, John, John Dowding, Elizabeth E. Shriberg, and Patti Price. 1993. A system for labeling self-repairs in speech. Technical Note 522, SRI International. find
  8. Bear, John and Patti Price. 1990. Prosody, syntax, and parsing. In Proceedings of the 28th Annual Meeting, pages 17-22. Association for Computational Linguistics. find
  9. Black, Ezra, Fred Jelinek, John Lafferty, David Magerman, Robert Mercer, and Salim Roukos. 1992. Towards history-based grammars: Using richer models for probabilistic parsing. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 134-139. Morgan Kaufman.
  10. Blackmer, Elizabeth R. and Janet L. Mitton. 1991. Theories of monitoring and the timing of repairs in spontaneous speech. Cognition, 39:173-194. find
  11. Breiman, Leo, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Wadsworth & Brooks. find
  12. Brown, Gillian and George Yule. 1983. Discourse Analysis. Cambridge University Press. find
  13. Brown, Peter F., Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  14. Charniak, Eugene, C. Hendrickson, N. Jacobson, and M. Perkowitz. 1993. Equations for part-of-speech tagging. In Proceedings of the National Conference on Artificial Intelligence, pages 784-789. find
  15. Chow, Yen-Lu and Richard Schwartz. 1989. The n-best algorithm: An efficient procedure for finding top n sentence hypotheses. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 199-202. find
  16. Church, Kenneth. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Proceedings of the 2nd Conference on Applied Natural Language Processing, pages 136-143.
  17. Dowding, John, Jean M. Gawron, Doug Appelt, John Bear, Lynn Cherny, Robert Moore, and Douglas Moran. 1993. Gemini: A natural language system for spoken-language understanding. In Proceedings of the 31st Annual Meeting, pages 54-61. Association for Computational Linguistics. ↗
  18. Entropic Research Laboratory, Inc. 1994. Aligner Reference Manual. Version 1.3. find
  19. Godfrey, John J., Edward C. Holliman, and Jane McDaniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of the International Conference on Audio, Speech and Signal Processing, pages 517-520. find
  20. Heeman, Peter A. 1997. Speech Repairs, Intonational Boundaries and Discourse Markers: Modeling Speakers' Utterances in Spoken Dialogue. Doctoral dissertation, Department of Computer Science, University of Rochester. find
  21. Heeman, Peter A. 1999. POS tags and decision trees for language modeling. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 129-137. find
  22. Heeman, Peter A. and James F. Allen. 1994. Detecting and correcting speech repairs. In Proceedings of the 32nd Annual Meeting, pages 295-302. Association for Computational Linguistics. find
  23. Heeman, Peter A. and James F. Allen. 1995. The Trains Spoken Dialog Corpus. CD-ROM, Linguistics Data Consortium. find
  24. Heeman, Peter A. and K. H. Loken-Kim. 1999. Detecting and correcting speech repairs in Japanese. In Proceedings of the ICPhS Satellite Meeting on Disfluency in Spontaneous Speech, pages 43-46. find
  25. Heeman, Peter A., K. H. Loken-Kim, and James F. Allen. 1996. Combining the detection and correction of speech repairs. In Proceedings of the 4th International Conference on Spoken Language Processing, pages 358-361. find
  26. Hindle, Donald. 1983. Deterministic parsing of syntactic non-fluencies. In Proceedings of the 21st Annual Meeting, pages 123-128. Association for Computational Linguistics. find
  27. Hirschberg, Julia and Diane Litman. 1993. Empirical studies on the disambiguation of cue phrases. Computational Linguistics, 19(3):501-530.
  28. Jelinek, Frederick. 1985. Self-organized language modeling for speech recognition. Technical report, IBM T.J. Watson Research Center. find
  29. Jelinek, Frederick. and Robert L. Mercer. 1980. Interpolated estimation of Markov source parameters from sparse data. In Proceedings of the Workshop on Pattern Recognition in Practice, pages 381-397. find
  30. Katz, Slava M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3):400-401. ↗
  31. Kikui, Gen-ichiro and Tsuyoshi Morimoto. 1994. Similarity-based identification of repairs in Japanese spoken language. In Proceedings of the 3rd International Conference on Spoken Language Processing, pages 915-918. find
  32. Kompe, Ralf, Andreas Kiet~ling, Heinrich Niemann, Elmar NOth, E. Gi~nter Schukat-Talamazzini, A. Zottmann, and Anton Batliner. 1995. Prosodic scoring of word hypotheses graphs. In Proceedings of the 4th European Conference on Speech Communication and Technology, pages 1333-1336. find
  33. Levelt, Willem J. 1983. Monitoring and self-repair in speech. Cognition, 14:41-104. ↗
  34. Lickley, Robin J. and Ellen G. Bard. 1992. Processing disfluent speech: Recognizing disfluency before lexical access. In Proceedings of the 2nd International Conference on Spoken Language Processing, pages 935-938. find
  35. Lickley, Robin J., Richard C. Shillcock, and Ellen G. Bard. 1991. Processing disfluent speech: How and when are disfluencies found? In Proceedings of the 2nd European Conference on Speech Communication and Technology, pages 1499-1502. find
  36. Litman, Diane J. 1996. Cue phrase classification using machine learning. Journal of Arti~cial Intelligence Research, 5:53-94. ↗
  37. MADCOW. 1992. Multi-site data collection for a spoken language corpus. In Proceedings of the DARPA Workshop on Speech and Natural Language Processing, pages 7-14. find
  38. Magerman, David M. 1994. Natural Language Parsing as Statistical Pattern Recognition. Doctoral dissertation, Department of Computor Science, Stanford University. ↗
  39. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330. ↗
  40. Martin, J. G. and W. Strange. 1968. The perception of hesitation in spontaneous speech. Perception and Psychophysics, 53:1-15. ↗
  41. Mast, Marion, Ralf Kompe, Stefan Harbeck, Andreas Kiet~ling, Heinrich Niemann, Elmar NOth, E. Giinther Schukat-Talamazzini, and Volker Warnke. 1996. Dialogue act classification with the help of prosody. In Proceedings of the 4th International Conference on Spoken Language Processing, pages 1728-1731. find
  42. Meteer, Marie and Rukmini Iyer. 1996. Modeling conversational speech for speech recognition. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 33-47. find
  43. Nakatani, Christine H. and Julia Hirschberg. 1994. A corpus-based study of repair cues in spontaneous speech. Journal of the Acoustical Society of America, 95(3):1603-1616. ↗
  44. Nooteboom, S. G. 1980. Speaking and unspeaking: Detection and correction of phonological and lexical errors. In Victoria A. Fromkin, editor, Errors in Linguistic Performance: Slips of the Tongue, Ear, Pen, and Hand. Academic Press, pages 86-97. find
  45. O'Shaughnessy, Douglas. 1994. Correcting complex false starts in spontaneous speech. In Proceedings of the International Conference on Audio, Speech and Signal Processing, pages 349-352. find
  46. Ostendorf, Marl, Colin Wightman, and Nanette Veilleux. 1993. Parse scoring with prosodic information: An analysis/synthesis approach. Computer Speech and Language, 7(2):193-210. ↗
  47. Schiffrin, Deborah. 1987. Discourse Markers. Cambridge University Press. find
  48. Shriberg, Elizabeth E. 1994. Preliminaries to a Theory of Speech Disfluencies. Doctoral dissertation, University of California at Berkeley. find
  49. Shriberg, Elizabeth E., Rebecca Bates, and Andreas Stolcke. 1997. A prosody-only decision-tree model for disfluency detection. In Proceedings of the 5th European Conference on Speech Communication and Technology, pages 2383-2386. find
  50. Shriberg, Elizabeth E. and Robin J. Lickley. 1993. Intonation of clause-internal filled pauses. Phonetica, 50(3):172-179. ↗
  51. Silverman, Ken, Mary Beckman, John Pitrelli, Mari Ostendorf, Colin Wightman, Patti Price, Janet Pierrehumbert, and Julia Hirschberg. 1992. ToBI: A standard for labelling English prosody. In Proceedings of the 2nd International Conference on Spoken Language Processing, pages 867-870. find
  52. Siu, Man-hung and Marl Ostendorf. 1996. Modeling disfluencies in conversational speech. In Proceedings of the 4th International Conference on Spoken Language Processing, pages 382-391. find
  53. Stolcke, Andreas and Elizabeth E. Shriberg. 1996a. Automatic linguistic segmentation of conversational speech. In Proceedings of the 4th International Conference on Spoken Language Processing, pages 1001-1004. find
  54. Stolcke, Andreas and Elizabeth E. Shriberg. 1996b. Statistical language modeling for speech disfluencies. In Proceedings of the International Conference on Audio, Speech and Signal Processing, pages 405-408. find
  55. Traum, David R. and Peter A. Heeman. 1997. Utterance units in spoken dialogue. In Elisabeth Maier, Marion Mast, and Susann LuperFoy, editors, Dialogue Processing in Spoken Language Systems, Lecture Notes in Artificial Intelligence. Springer-Verlag, Heidelberg, pages 125-140. find
  56. Wang, Michelle Q. and Julia Hirschberg. 1992. Automatic classification of intonational phrase boundaries. Computer Speech and Language, 6:175-196. find
  57. Wightman, Colin and Mari Ostendorf. 1994. Automatic labeling of prosodic patterns. IEEE Transactions on Speech and Audio Processing, 2(4):469-481. ↗
  58. Zeppenfeld, Torsten, Michael Finke, Klaus Ries, Martin Westphal, and Alex Waibel. 1997. Recognition of conversational telephone speech using the Janus speech engine. In Proceedings of the International Conference on Audio, Speech and Signal Processing, pages 1815-1818. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/heeman-1999-speech-repains-intonational-phrases-discourse/