Elamur Research⌕
☰
home / papers / 2016 / Multi-domain Neural Network Language Generation for Spoken Dialogue Systems

Multi-domain Neural Network Language Generation for Spoken Dialogue Systems

Paper · 2016 · Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić et al.

Summary

The paper addresses how to build a natural language generator for a new spoken dialogue domain when only a small amount of in-domain data is available. The underlying difficulty is that the space of possible semantic inputs grows very quickly as domains are added. The authors build on the SC-LSTM generator, an LSTM whose extra reading gate controls how the dialogue-act features are consumed. They propose a two-step adaptation recipe:

  1. Data counterfeiting. Source-domain examples are delexicalised, and their slots are swapped for target-domain slots of the same functional class. This produces synthetic target data for pre-training.
  2. Discriminative training (DT). The model is then fine-tuned on a small in-domain set with a discriminative objective that rewards high sentence BLEU and penalises slot errors.

The recipe was tested on adaptation between four dialogue domains, including laptop↔TV. With about 10% of the target data, adapted models came close to models trained from scratch on the full data, both on BLEU and slot error rate and in human ratings. The approach matters because it lowers the cost of extending data-driven generators to new domains.

Key points

  • Base model: SC-LSTM, trained with cross-entropy plus a regulariser on how the dialogue-act vector changes over time (η = 10⁻⁴, ξ = 100).
  • Counterfeiting step: slots are grouped into three functional classes (informable, requestable, binary). Source slots and values missing from the target domain are replaced with target ones of the same class.
  • DT setup:
    • The objective combines sentence BLEU (weight 1.0) and slot error rate (weight −1.0), with γ = 5.0.
    • The generator samples 50 candidates per dialogue act, and duplicates are removed.
    • In the laptop→TV setting (10% ≈ 700 examples), slot error rate dropped to near zero after DT.
  • Human evaluation setup: run on AMT with roughly 2000 dialogue acts per scenario. Each system produced 5 realisations per act, rated for informativeness and naturalness on a 3-point scale.
  • Human evaluation results:
    • On TV→laptop, scores (informativeness / naturalness) were: scrALL 2.64 / 2.37, DT-10% 2.52 / 2.25, scr-10% 2.24 / 2.03.
    • On laptop→TV, DT-10% informativeness (2.51) was statistically indistinguishable from scrALL (2.54).
    • In pairwise preference tests, judges favoured DT-10% over ML-10% in both scenarios (55.1% and 51.9%), but the difference was not significant.

Interesting ideas

  • I1. Synthesising target data by slot swapping. Counterfeiting maps source slots onto target slots of the same functional type. This is a simple alternative to complex weight-tying schemes and could bootstrap generators or taggers for any schema-driven task where a new domain shares structure with an old one.
  • I2. Delexicalisation as a transfer enabler. Replacing concrete values with slot tokens lets the model learn sentence templates that are independent of the domain. It is a cheap preprocessing choice that makes cross-domain reuse much easier.
  • I3. Discriminative fine-tuning on sampled hypotheses. Sampling many outputs, removing duplicates, and pushing the model toward candidates that score well on task metrics makes small data go further. This could suit low-resource generation settings, for example a new language or genre.
  • I4. Optimising a metric that is easy to compute exactly. Slot error rate was much easier to drive down than BLEU, likely because sentence BLEU only approximates corpus BLEU. When choosing training rewards, it helps to prefer metrics whose sentence-level form matches the metric used for evaluation.
  • I5. Evaluating output variety, not just the top output. Judges rated several sampled realisations per input, and identical outputs from both systems were filtered out. This tests a system's diversity rather than its single best answer, and the filtering saves annotator effort.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksText generation · 145 works

Cite

Cite as Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke and Steve Young. 2016. Multi-domain Neural Network Language Generation for Spoken Dialogue Systems. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1015

@inproceedings{wen2016multi,
  title = {{Multi-domain Neural Network Language Generation for Spoken Dialogue Systems}},
  author = {Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke and Steve Young},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1015},
  url = {https://doi.org/10.18653/v1/n16-1015}
}

References

35 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Auli and Jianfeng Gao. 2014. Decoder integration and expected bleu training for recurrent neural network language models. In Proceedings of ACL. Association for Computational Linguistics. find
  2. Michael Auli, Michel Galley, and Jianfeng Gao. 2014. Large-scale expected bleu training of phrase-based reordering models. In Proceedings of EMNLP. Association for Computational Linguistics. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicol… find
  3. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. find
  4. Jerome R. Bellegarda. 2004. Statistical language model adaptation: review and perspectives. Speech Communication. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and… find
  5. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL. Association for Computational Linguistics.
  6. Dan Bohus and Alexander I. Rudnicky. 2009. The ravenclaw dialog management framework: Architecture and systems. find
  7. Computer Speech and Language. Xie Chen, Tan Tian, Liu Xunying, Lanchantin Pierre, Wan Moquan, Mark Gales, and Woodland Phil. 2015. Recurrent neural network language model adaptation for multi-genre broadcast speech recognition. In Proceedings of InterSpeech. find
  8. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of EMNLP. Association for Computational Linguistics.
  9. Heriberto Cuayhuitl, Nina Dethlefs, Helen Hastie, and Xingkun Liu. 2014. Training a statistical surface realiser from automatic slot labelling. In Spoken Language Technology Workshop (SLT), 2014 IEEE. Hal Daumé III. 2009. Frustratingly easy domain adaptation. CoRR, abs/0907.1815. — [arXiv:0907.1815](https://arxiv.org/abs/0907.1815) find
  10. Lixin Duan, Dong Xu, and Ivor W. Tsang. 2012. Learning with augmented features for heterogeneous domain adaptation. CoRR, abs/1206.4660. Milica Gašić, Nikola Mrkšić, Pei-hao Su, David Vandyke, Tsung-Hsien Wen, and Steve J. Young. 2015. Policy committee for adaptation in multi-domain spoken dialo… — [arXiv:1206.4660](https://arxiv.org/abs/1206.4660) find
  11. Jean-Luc Gauvain and Chin-Hui Lee. 1994. Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains. Speech and Audio Processing, IEEE Transactions on. find
  12. Daniel Gildea and Thomas Hofmann. 1999. Topic-based language models using em. In Proceedings of EuroSpeech. find
  13. Xiaodong He and Li Deng. 2012. Maximum expected bleu training of phrase and lexicon translation models. In Proceedings of ACL. Association for Computational Linguistics. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. find
  14. Neural Computation. Deirdre Hogan, Jennifer Foster, Joachim Wagner, and Josef van Genabith. 2008. Parser-based retraining for domain adaptation of probabilistic generators. In Proceedings of INLG. Association for Computational Linguistics. find
  15. Amy Isard, Carsten Brockmann, and Jon Oberlander. 2006. Individuality and alignment in generated dialogues. In Proceedings of INLG. Association for Computational Linguistics. find
  16. Ronald M. Kaplan and Joan Bresnan. 1982. Lexical-Functional Grammar: a formal system for grammatical representation. In Joan Bresnan, editor, The mental representation of grammatical relations. find
  17. MIT Press. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of StatMT. Association for Computational Linguistics.
  18. Hong-kwang Kuo, Eric Fosler-lussier, Hui Jiang, and Chin-hui Lee. 2002. Discriminative training of language models for speech recognition. In Proceedings of ICASSP. find
  19. Irene Langkilde and Kevin Knight. 1998. Generation that exploits corpus-based statistical knowledge. In Proceedings of ACL. Association for Computational Linguistics.
  20. Chris Leggetter and Philip Woodland. 1995. Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models. find
  21. Computer Speech and Language. Oliver Lemon. 2008. Adaptive natural language generation in dialogue using reinforcement learning. In Proceedings of SemDial. find
  22. Franois Mairesse and Marilyn Walker. 2008. Trainable generation of big-five personality styles through data-driven parameter estimation. In Proceedings of ACL. Association for Computational Linguistics. François Mairesse and Marilyn A. Walker. 2011. Controlling user perceptions of linguistic style:… find
  23. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Trans. on Knowledge and Data Engineering. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL. Association for Computational Linguistics.
  25. Yangyang Shi, Martha Larson, and Catholijn M. Jonker. 2015. Recurrent neural network language model adaptation with curriculum learning. find
  26. Computer, Speech and Language. Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of ACL. Association for Computational Linguistics. find
  27. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Proceedings of CICLing 2005. find
  28. Volker Tresp. 2000. A bayesian committee machine. find
  29. Neural Computation. Paul Voigtlaender, Patrick Doetsch, Simon Wiesler, Ralf Schluter, and Hermann Ney. 2015. Sequence-discriminative training of recurrent neural networks. In Proceedings of ICASSP. find
  30. Marilyn A Walker, Owen C Rambow, and Monica Rogati. 2002. Training a sentence planner for spoken dialogue using boosting. find
  31. Computer Speech and Language. Marilyn Walker, Amanda Stent, Franois Mairesse, and Rashmi Prasad. 2007. Individual and domain adaptation in sentence planning for dialogue. Journal of Artificial Intelligence Research (JAIR). find
  32. Wayne Ward and Sunil Issar. 1994. Recent improvements in the cmu spoken language understanding system. In Proceedings of Workshop on HLT. Association for Computational Linguistics. find
  33. Tsung-Hsien Wen, Hung-Yi Lee, Tai-Yuan Chen, and Lin-Shan Lee. 2012. Personalized language modeling by crowd sourcing with social network data for voice access of cloud applications. In Proceedings of SLT. Tsung-Hsien Wen, Aaron Heidel, Hung yi Lee, Yu Tsao, and Lin-Shan Lee. 2013. Recurrent neural… find
  34. Paul J Werbos. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE. Steve Young, Milica Gašić, Blaise Thomson, and Jason D. Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE. find
  35. Dong Yu, Kaisheng Yao, Hang Su, Gang Li, and Frank Seide. 2013. Kl-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition. In Proceedings of ICASSP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wen-2016-multi-domain-neural-network-language/