Elamur Research⌕
☰
home / papers / 2001 / Edit Detection and Parsing for Transcribed Speech

Edit Detection and Parsing for Transcribed Speech

Paper · 2001 · Eugene Charniak and Mark Johnson

Summary

Parsing transcribed conversation is harder than parsing written text. A major reason is speech repairs, where a speaker abandons a stretch of words and restarts. Charniak and Johnson propose a two-stage pipeline. First, a classifier labels each word as edited (corrected by a restart or repair) or not, and the edited words are removed. Then an unchanged statistical parser (the Charniak parser), trained on Switchboard speech, parses what remains. For evaluation only, the removed words are put back into the tree.

The edit detector is a linear classifier learned with Collins' greedy boosting. Its features rely heavily on "rough copies", which are repeated word and POS-tag sequences that often signal repairs. The detector lowers the edited-word error rate from 5.9% (the trivial baseline) to 2.2%.

The authors also introduce a relaxed metric that does not penalize the exact tree attachment of EDITED nodes. Under this metric the full system reaches 85.3% precision and 86.5% recall. The work shows that shallow preprocessing can make an off-the-shelf parser usable on disfluent speech.

Key points

  • Data: the parsed Switchboard corpus from the Linguistic Data Consortium. Repairs appear there as EDITED nodes, and as [ + ] disfluency tags in the POS-tagged version.

  • Edit detector design:

    • Each word token is classified on its own; punctuation takes the label of the preceding word.
    • The classifier uses 18 conditioning variables and roughly 10^6 candidate features.

  • Training procedure:

    • Boosting runs for 25,000 iterations, and the model kept is the one with the lowest error on the development set.
    • That minimum usually comes around iteration 12,000, with about 8,000 nonzero weights.

  • Parsing results (labeled precision / recall):

    SettingPrecisionRecall
    Gold EDITED positions87.888.1
    Gold tags fed to the detector85.486.6
    Machine tags fed to the detector85.386.5
    The same parser on WSJ text89.589.6

  • Comparison with prior work: the closest study, Stolcke et al. on Switchboard, reports 2.6% error against a 4.5% baseline. The authors warn that the different baselines make a direct comparison doubtful.

Interesting ideas

  • I1. Handle only the truly new phenomenon in preprocessing. The authors reason that filled pauses, parentheticals and ungrammatical sentences can be learned by a statistical parser trained on in-domain data. Only speech repairs get special treatment. This kind of triage keeps a pipeline simple and lets existing models be reused unchanged.
  • I2. "Rough copy" pattern as a feature source. The pattern is a source span, optional free-final words such as partial words or "and", an optional interregnum such as "uh" or "I mean", and then a span with the same tag sequence. Defining such structured repetition patterns could help detect disfluencies in other languages, including Ukrainian, or clean up noisy ASR output.
  • I3. Relaxed evaluation metric. The metric ignores where an EDITED node attaches, much as earlier metrics ignored punctuation attachment. This avoids penalizing arbitrary annotation choices, and the same idea fits any task where some annotation decisions are conventional rather than meaningful.
  • I4. Removing duplicate features cheaply. Features that always take the same values on the training data are found by hashing XORed random bitmaps, and only the simplest one in each group is kept. This is a cheap trick for shrinking very large sparse feature spaces.
  • I5. Reading the precision/recall gap as a diagnostic. Precision falling below recall is unusual for parsers. The authors trace it to the edit detector missing some edited words, which leaves extra words and therefore extra constituents in the parser output. Such asymmetries can point to which pipeline stage is causing errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brown UniversityProvidence · United States58 works · 25,310 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Eugene Charniak and Mark Johnson. 2001. Edit Detection and Parsing for Transcribed Speech. Second Meeting of the North American Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{charniak2001edit,
  title = {{Edit Detection and Parsing for Transcribed Speech}},
  author = {Eugene Charniak and Mark Johnson},
  booktitle = {Second Meeting of the North American Chapter of the Association for Computational Linguistics},
  year = {2001},
  doi = {10.3115/1073336.1073352},
  url = {https://doi.org/10.3115/1073336.1073352}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bear, J., Dowding, J. and Shriberg, E. Integrating multiple knowledge sources for detection and correction of repairs in human-computer dialog. In Proceedings of the 30th Annual Meeting of the Association for Computational Linguistics. 56–63. find
  2. Charniak, E. Statistical parsing with a context-free grammar and word statistics. In Proceedings of the Fourteenth National Conference on Artificial Intelligence. AAAI Press/MIT Press, Menlo Park, CA, 1997, 598–603. find
  3. Charniak, E. A maximum-entropy-inspired parser. In Proceedings of the 2000 Conference of the North American Chapter of the Association for Computational Linguistics. ACL, New Brunswick NJ, 2000.
  4. Collins, M. J. A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Annual Meeting of the ACL. 1996. find
  5. Collins, M. J. Three generative lexicalized models for statistical parsing. In Proceedings of the 35th Annual Meeting of the ACL. 1997, 16–23. find
  6. Collins, M. J. Head-Driven Statistical Models for Natural Language Parsing. University of Pennsylvania, Ph.D. Dissertation, 1999. find
  7. Collins, M. J. Discriminative reranking for natural language parsing. In Proceedings of the International Conference on Machine Learning (ICML 2000). 2000.
  8. Core, M. G. and Schubert, L. K. A syntactic framework for speech repairs and other disruptions. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics. 1999, 413–420. find
  9. Heeman, P. A. and Allen, J. F. Intonational boundaries, speech repairs and discourse markers: modeling spoken dialog. In 35th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics. 1997, 254–261. find
  10. Heeman, P. A. and Allen, J. F. Speech repairs, intonational phrases and discourse markers: modeling speakers’ utterances in spoken dialogue. Computational Linguistics 254 (1999). find
  11. Hindle, D. Deterministic parsing of syntactic non-fluencies. In Proceedings of the 21st Annual Meeting of the Association for Computational Linguistics. 1983, 123–128. find
  12. Magerman, D. M. Statistical decision-tree models for parsing. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics. 1995, 276–283.
  13. Nakatani, C. H. and Hirschberg, J. A corpus-based study of repair cues in spontaneous speech. Journal of the Acoustical Society of America 953 (1994), 1603–1616. find
  14. Ratnaparkhi, A. Learning to parse natural language with maximum entropy models. Machine Learning 34 1/2/3 (1999), 151–176. find
  15. Shriberg, E. E. Preliminaries to a Theory of Speech Disfluencies. In PhD Dissertation. Department of Psychology, University of California-Berkeley, 1994. find
  16. Singer, Y. and Schapire, R. E. Improved boosting algorithms using confidence-based predictions. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory. 1998, 80–91. find
  17. Stolcke, A. and Shriberg, E. Automatic linguistic segmantation of conversational speech. In Proceedings of the 4th International Conference on Spoken Language Processing (ICSLP-96). 1996. find
  18. Stolcke, A., Shriberg, E., Bates, R., Ostendorf, M., Hakkani, D., Plauche, M., Tür, G. and Lu, Y. Automatic detection of sentence boundaries and disfluencies based on recognized words. Proceedings of the International Conference on Spoken Language Processing 5 (1998), 2247–2250. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/charniak-2001-edit-detection-parsing-transcribed-speech/