Elamur Research⌕
☰
home / papers / 2004 / Combining Acoustic and Pragmatic Features to Predict Recognition Performance in Spoken Dialogue Systems

Combining Acoustic and Pragmatic Features to Predict Recognition Performance in Spoken Dialogue Systems

Summary

Spoken dialogue systems must decide whether to accept, reject, or ignore each incoming recognition hypothesis, and sometimes whether to ask for clarification. Wrong choices either cause misunderstandings or make the system overly cautious. Gabsdil and Lemon tackle this with machine learning. Their classifiers combine acoustic confidence scores with pragmatic plausibility features drawn from dialogue context. They apply these classifiers to n-best hypotheses rather than only the top one, then pick a best candidate.

The data come from a user study with the WITAS multimodal system for commanding a simulated UAV, which uses the Information State Update (ISU) approach. The authors compare TiMBL and RIPPER against a grammar-switching baseline that relies on fixed rejection thresholds. With optimized parameters, TiMBL raises weighted f-score by about 25% over the baseline. This result comes close to an oracle upper bound.

The work matters because it shows that context-derived features can substantially improve how a dialogue system reacts to user input. The authors also argue the method should carry over to other ISU-based systems.

Key points

  • Data: 6 subjects completed 5 tasks each with the WITAS system (Nuance 8.0 recognizer, Gemini grammar), giving 30 dialogues and 303 utterances. Each utterance was labelled in-grammar, out-of-grammar, or crosstalk.
  • Baseline: the grammar-switching system reached 65.68% accuracy and 61.81% weighted f-score. Its main weakness was accepting too many utterances.
  • Best result: TiMBL with optimized parameters, using recognition, utterance, dialogue, and task features (REC+UTT+DIAL+TASK), on 10-best output:
    • 86.14% accuracy and 86.39% weighted f-score for 3 classes;
    • 82.51% accuracy and 83.29% weighted f-score for 4 classes (including clarify).
  • Upper bound: the oracle scored about 94%, because the 10-best lists of 18 in-grammar utterances contained no correct logical form. A χ² test found no significant difference between the oracle and optimized TiMBL.
  • Tuning effect: parameter optimization added roughly 9% over default TiMBL settings, and all learned classifiers significantly outperformed the baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1218955.1218999

@inproceedings{gabsdil2004combining,
  title = {{Combining Acoustic and Pragmatic Features to Predict Recognition Performance in Spoken Dialogue Systems}},
  author = {Gabsdil, Malte and Lemon, Oliver},
  booktitle = {Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)},
  year = {2004},
  doi = {10.3115/1218955.1218999},
  url = {https://doi.org/10.3115/1218955.1218999}
}

References

15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Boros, W. Eckert, F. Gallwitz, G. Görz, G. Han-rieder, and H. Niemann. 1996. Towards Understanding Spontaneous Speech: Word Accuracy vs. Concept Accuracy. In Proc. ICSLP-96. find
  2. Ananlada Chotimongkol and Alexander I. Rudnicky. 2001. N-best Speech Hypotheses Reordering Using Linear Regression. In Proceedings of EuroSpeech 2001, pages 1829–1832. find
  3. William W. Cohen. 1995. Fast Effective Rule Induction. In Proceedings of the 12th International Conference on Machine Learning. find
  4. EC FP6 IST-507802, http://www.talk-project.org Walter Daelemans and Véronique Hoste. 2002. Evaluation of Machine Learning Methods for Natural Language Processing Tasks. In Proceedings of LREC-02. ↗
  5. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2002. TIMBL: Tilburg Memory Based Learner, version 4.2, Reference Guide. In ILK Technical Report 02-01. find
  6. John Dowding, Jean Mark Gawron, Doug Appelt, John Bear, Lynn Cherny, Robert Moore, and Douglas Moran. 1993. GEMINI: a natural language system for spoken-language understanding. In Proceedings of ACL-93. find
  7. Malte Gabsdil. 2003. Classifying Recognition Results for Spoken Dialogue Systems. In Proceedings of the Student Research Workshop at ACL-03. find
  8. Perry R. Hinton. 1995. Statistics Explained – A Guide For Social Science Students. Routledge. find
  9. Oliver Lemon and Alexander Gruenstein. 2004. Multithreaded context for robust conversational interfaces: context-sensitive speech recognition and interpretation of corrective fragments. ACM Transactions on Computer-Human Interaction. (to appear). find
  10. Oliver Lemon, Alexander Gruenstein, and Stanley Peters. 2002. Collaborative activities and multi-tasking in dialogue systems. Traitement Automatique des Langues, 43(2):131–154. find
  11. Oliver Lemon. 2004. Context-sensitive speech recognition in ISU dialogue systems: results for the grammar switching approach. In Proceedings of the 8th Workshop on the Semantics and Pragmatics of Dialogue, CATALOG’04. find
  12. Diane J. Litman, Julia Hirschberg, and Marc Swerts. 2000. Predicting Automatic Speech Recognition Performance Using Prosodic Cues. In Proceedings of NAACL-00. find
  13. Erwin Marsi, Martin Reynaert, Antal van den Bosch, Walter Daelemans, and Véronique Hoste. 2003. Learning to predict pitch accents and prosodic boundaries in Dutch. In Proceedings of ACL-03. find
  14. David Traum, Johan Bos, Robin Cooper, Staffan Larsson, Ian Lewin, Colin Matheson, and Massimo Poesio. 1999. A Model of Dialogue Moves and Information State Revision. Technical Report D2.1, Trindi Project. find
  15. Marilyn Walker, Jerry Wright, and Irene Langkilde. 2000. Using Natural Language Processing and Discourse Features to Identify Understanding Errors in a Spoken Dialogue System. In Proceedings of ICML-2000. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/gabsdil-2004-combining-acoustic-pragmatic-features-predict/