Elamur Research⌕
☰
home / papers / 2013 / Impact of ASR N-Best Information on Bayesian Dialogue Act Recognition

Impact of ASR N-Best Information on Bayesian Dialogue Act Recognition

Paper · 2013 · Heriberto Cuayáhuitl, Nina Dethlefs, Helen Hastie et al.

Summary

The paper addresses mapping noisy spoken input to the correct user dialogue act in a slot-filling system. Instead of building a new recogniser, the authors add a re-ranking step after an existing one, the Let's Go parser. The re-ranker rescores the parser's hypotheses of the form DialogueActType(attribute-value pairs). It uses several Bayesian networks: one for the act type, one for the slot, and one for each slot's values. A hypothesis is scored by multiplying these factors and normalising. The features describe what the system said in recent turns, which keywords were spotted in the user turn, and which slot values appear in the ASR N-best list. On a subset of the Let's Go bus-information corpus, adding the N-best evidence raised accuracy from 66.9% to 73.9%. This beat both a semi-random choice among the parser's hypotheses and the original Let's Go recogniser. Accuracy was about the same whether the model saw one preceding system act or more. Recognition can therefore happen early without losing quality.

Key points

  • Data: 779 Let's Go dialogues (7275 turns) recorded in summer 2010. The split was 70% training and 30% test, repeated over 100 random splits. The domain has 12 system act types, 11 user act types (silence is modelled explicitly) and 5 main slots.
  • Features: 43 discrete variables in total. These are 25 binary system-side features, 17 user-side features (keywords plus a binned duration) and one word-level * nbest variable holding slot values from the ASR N-best list.
  • Fourteen Bayesian classifiers were trained. Structure was learned with K2 (Weka), parameters with maximum likelihood, and inference used the Junction tree algorithm (JavaBayes).
  • Accuracy results: IncK reached 73.9% with N-best evidence and 66.9% without it. The semi-random baseline scored 62.9% and the Let's Go recogniser 69%. Differences were significant at p < 0.05 (two-sided Wilcoxon Signed-Rank test).
  • Differences between context sizes Inc_i (i > 0) were not significant. The authors traced over half of the remaining errors to environmental noise or user behaviour that corrupted ASR output.

Interesting ideas

  • I1. The structured output (act type, slot, value) is split into separate small Bayesian networks whose scores are multiplied, which avoids one huge joint table (their example joint would need about 10^11 parameters) and keeps each part trainable on modest data.
  • I2. To measure how much the N-best feature contributes, they kept that variable in the network but replaced its learned distribution with a uniform one, so the ablated model keeps the same structure and only the information is removed.
  • I3. Results are bracketed between a semi-random pick from the upstream candidate list and an oracle Ceiling that always picks the correct candidate if present, so the re-ranker's gain is judged relative to what the candidates allow.
  • I4. Silence or "no real act" is an explicit output class, because the recogniser otherwise forces some interpretation onto noise.
  • I5. They tested how much prior context is actually needed and found that one preceding system act was as good as several.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A speech-act or frame hypothesis could be scored as a product of independent factors for act type, role and filler. Each factor stays small, and the explanation lists each factor's contribution.
  • I2 (learn): This works as a negative control in evaluation. Flatten one input, such as a witness parser's vote or a lexical cue, to uniform, and check whether accuracy really drops.
  • I3 (tree): When choosing among candidate trees from several parsers, report the random-pick and oracle-in-list scores alongside the chooser. This separates ranking quality from candidate-generation quality.
  • I4 (frames): The speech-act layer could include an explicit "no act / not addressed to the reader" label. Then fragments, asides or garbled text are not forced into an assertion or question frame.
  • I5 (frames): Before building long-range discourse features for speech-act or implicature decisions, measure accuracy with only the immediately preceding utterance. A small, explainable context may be enough.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heriot-Watt UniversityEdinburgh · United Kingdom10 works · 5,081 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 3 Dialogue systems · 401 works Speech & spoken language · 233 works Speech act theory · 160 works

Cite

Cite as Heriberto Cuayáhuitl, Nina Dethlefs, Helen Hastie and Oliver Lemon. 2013. Impact of ASR N-Best Information on Bayesian Dialogue Act Recognition. Proceedings of the SIGDIAL 2013 Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cuayahuitl2013impact,
  title = {{Impact of ASR N-Best Information on Bayesian Dialogue Act Recognition}},
  author = {Heriberto Cuayáhuitl, Nina Dethlefs, Helen Hastie and Oliver Lemon},
  booktitle = {Proceedings of the SIGDIAL 2013 Conference},
  year = {2013},
  url = {https://aclanthology.org/W13-4047/}
}

References

20 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA. find
  2. D. Bohus and A. Rudnicky. 2006. A k hypotheses + other” belief updating model. In AAAI Workshop on Statistical and Empirical Approaches to Spoken Dialogue Systems. find
  3. H. Cuayáhuitl and I. Kruijff-Korbayová. 2011. Learning human-robot dialogue policies combining speech and visual beliefs. In IWSDS, pages 133–140. find
  4. Nina Dethlefs and Heriberto Cuayáhuitl. 2011. Combining Hierarchical Reinforcement Learning and Bayesian Networks for Natural Language Generation in Situated Dialogue. In ENLG, Nancy, France. find
  5. Björn Gambäck, Fredrik Olsson, and Oscar Täckström. 2011. Active Learning for Dialogue Act Classification. In INTERSPEECH, pages 1329–1332. find
  6. Sergio Grau, Emilio Sanchis, Maria Jose Castro, and David Vilar. 2004. Dialogue Act Classification Using a Bayesian Approach. In SPECOM. find
  7. Matthew Henderson, Milica Gasic, Blaise Thomson, Pirros Tsiakoulis, Kai Yu, and Steve Young. 2012. Discriminative spoken language understanding using word confusion networks. In SLT, pages 176–181. find
  8. Simon Keizer and Rieks op den Akker. 2007. Dialogue Act Recognition Under Uncertainty Using Bayesian Networks. Natural Language Engineering, 13(4):287–316. find
  9. Pierre Lison. 2012. Probabilistic dialogue models with prior domain knowledge. In SIGDIAL Conference, pages 179–188. find
  10. Neville Mehta, Rakesh Gupta, Antoine Raux, Deepak Ramachandran, and Stefan Krawczyk. 2010. Probabilistic ontology trees for belief tracking in dialog systems. In SIGDIAL Conference, pages 37–46. find
  11. Andrew Y. Ng and Michael I. Jordan. 2001. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In NIPS, pages 841–848. find
  12. Antoine Raux, Brian Langner, Dan Bohus, Alan W. Black, and Maxine Eskenazi. 2005. Let’s go public! Taking a Spoken Dialog System to the Real World. In INTERSPEECH, pages 885–888. find
  13. Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Shrikanth Narayanan. 2009. Combining Lexical, Syntactic and Prosodic Cues for Improved Online Dialog Act Tagging. Computer Speech & Language, 23(4):407–422. ↗
  14. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca A. Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialog Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, 26(3):339–373. find
  15. Dinoj Surendran and Gina-Anne Levow. 2006. Dialog Act Tagging with Support Vector Machines and Hidden Markov Models. In INTER-SPEECH. find
  16. David M. Tax, Martijn van Breukelen, Robert P. Duin, and Josef Kittler. 2000. Combining multiple classifiers by averaging or by multiplying? Pattern Recognition, 33(9):1475–1485, September. ↗
  17. Blaise Thomson, Jost Schatzmann, and Steve Young. 2008. Bayesian update of dialogue state for robust dialogue systems. In ICASSP, pages 4937–4940. find
  18. Jason D. Williams. 2007. Using particle filters to track dialogue state. In ASRU, pages 502–507. find
  19. H. Wright, Massimo Poesio, and Stephen Isard. 1999. Using high level dialogue information for dialogue act recognition using prosodic features. In Proceedings of an ESCA Tutorial and Research Workshop on Dialogue and Prosody, pages 139–143, Eindhoven, The Netherlands. find
  20. Matthias Zimmermann, Yang Liu, Elizabeth Shriberg, and Andreas Stolcke. 2005. Toward Joint Segmentation and Classification of Dialog Acts in Multiparty Meetings. In MLMI, pages 187–193. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cuayahuitl-2013-impact-asr-n-best-information/