Elamur Research⌕
☰
home / papers / 2016 / Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings

Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings

Paper · 2016 · Ondřej Dušek and Filip Jurčíček

Summary

The paper asks whether a statistical generator for spoken dialogue systems works better as a pipeline, with sentence planning followed by surface realization, or as a single joint step. The authors build one seq2seq generator (an LSTM encoder-decoder with attention) that can run in either mode. It takes a dialogue act and outputs either a deep syntax tree, which the TectoMT surface realizer then turns into text, or the sentence itself. Beam search produces candidates, and an RNN classifier reranks them by penalizing outputs that leave out or invent slot values. The system learns from unaligned pairs of meaning representations and sentences in the small BAGEL restaurant dataset. Both modes train successfully. Direct string generation gives the best results, beating earlier BLEU/NIST scores on BAGEL while producing fewer irrelevant details. This suggests that neural NLG can work without manual alignments and with far less data than comparable RNN generators used at the time.

Key points

  • Data: BAGEL contains 202 dialogue acts, each with two paraphrases. Names and similar values are delexicalized to "X". Evaluation uses 10-fold cross-validation (181 training and 21 test DAs per fold), with 10 training DAs held out for validation.
  • Pipeline mode: the generator outputs deep syntax trees made of lemmas and formemes in bracketed notation. Training trees come from automatic analysis with the Treex toolkit, and the TectoMT surface realizer turns trees into text.
  • Best results:
    • Strings with reranker (beam 100): BLEU 62.76, NIST 5.669, 19 semantic errors.
    • Trees with reranker: BLEU up to 60.93 (beam 10), with 19 errors at beam 100.
    • Earlier perceptron-based system of Dušek and Jurčíček (2015): BLEU 59.89, NIST 5.231, 30 errors.
  • Effect of decoding: greedy decoding is much weaker, especially for strings (BLEU 52.54, 37 errors). Semantic errors were counted manually on a sample of 42 outputs.
  • Code: the generator is released as TGen (github.com/UFAL-DSG/tgen) and implemented in TensorFlow.

Interesting ideas

  • I1. Semantic reranker via Hamming distance: a classifier predicts which DA types and slot-value pairs a candidate output expresses. The distance between that vector and the input's vector becomes a penalty on the candidate's log probability. Any data-to-text or template-filling task could use this cheap check against hallucinated or dropped content.
  • I2. Trees as token sequences: the generator emits trees as bracketed token sequences, while the reranker sees a flattened list of lemma-formeme pairs. This lets an ordinary sequence model produce structured output without a special tree decoder.
  • I3. Controlled pipeline vs. joint comparison: running both setups on the same architecture isolates the effect of the intermediate representation. The same design works for any question of the form "is an intermediate layer worth it?"
  • I4. Delexicalization against sparsity: replacing rare values such as names with placeholders lets a model learn from about two hundred examples. This is relevant for low-resource languages and small domain corpora.
  • I5. Separate validation and final decoding: model selection uses validation BLEU with greedy decoding and several random restarts, while beam search and reranking are added only at the end. This keeps validation fast and shows clearly how much each decoding component contributes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Charles UniversityPrague · Czechia108 works · 20,329 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSyntax & grammar formalisms · 161 works

Cite

Cite as Ondřej Dušek and Filip Jurčíček. 2016. Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-2008

@inproceedings{dusek2016sequence,
  title = {{Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings}},
  author = {Ondřej Dušek and Filip Jurčíček},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-2008},
  url = {https://doi.org/10.18653/v1/p16-2008}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. O. Dušek, L. Gomes, M. Novák, M. Popel, and R. Rosa. 2015. New Language Pairs in TectoMT. In Proceedings of the 10th Workshop on Machine Translation, pages 98–104, Lisbon, Portugal. Association for Computational Linguistics. find
  2. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B.… find
  3. A. Graves. 2013. Generating Sequences With Recurrent Neural Networks. arXiv:1308.0850 [cs], August. find
  4. S. Jean, K. Cho, R. Memisevic, and Y. Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lo…
  5. G. Angeli, P. Liang, and D. Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 502–512.
  6. D. Kingma and J. Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations. arXiv:1412.6980. find
  7. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473.
  8. P. Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP, pages 388–395.
  9. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3:1137– 1155.
  10. I. Konstas and M. Lapata. 2013. A Global Model for Concept-to-Text Generation. Journal of Artificial Intelligence Research, 48:305–346. find
  11. K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages… find
  12. F. Mairesse, M. Gašić, F. Jurčı́ček, S. Keizer, B. Thomson, K. Yu, and S. Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1552–1561. find
  13. N. Dethlefs, H. Hastie, H. Cuayáhuitl, and O. Lemon. 2013. Conditional Random Fields for Responsive Surface Realisation using Global Features. In Proceedings of ACL, Sofia. find
  14. H. Mei, M. Bansal, and M. R. Walter. 2015. What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment. arXiv:1509.00838 [cs], September.
  15. G. Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  16. D. S. Paiva and R. Evans. 2005. Empirically-based control of natural language generation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 58–65, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  17. O. Dušek and F. Jurčı́ček. 2015. Training a Natural Language Generator From Unaligned Data. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 451–461, Beijing, China. Associat… find
  18. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  19. M. Popel and Z. Žabokrtský. 2010. TectoMT: modular NLP framework. In Proceedings of IceTAL, 7th International Conference on Natural Language Processing,, pages 293–304, Reykjavı́k. find
  20. Ondřej Dušek, Zdeněk Žabokrtský, Martin Popel, Martin Majliš, Michal Novák, and David Mareček. 2012. Formemes in English-Czech Deep Syntactic MT. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 267–274, Montréal, Canada. Association for Computational Linguis…
  21. M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. 2015. Sequence Level Training with Recurrent Neural Networks. arXiv:1511.06732 [cs], November.
  22. S. Young, M. Gašić, S. Keizer, F. Mairesse, J. Schatzmann, B. Thomson, and K. Yu. 2010. The Hidden Information State model: A practical framework for POMDP-based spoken dialogue management. Computer Speech & Language, 24(2):150– 174, April. find
  23. A. Raux, B. Langner, D. Bohus, A. W. Black, and M. Eskenazi. 2005. Let’s go public! taking a spoken dialog system to the real world. In in Proc. of Interspeech 2005. Citeseer. find
  24. E. Reiter and R. Dale. 2000. Building Natural Language Generation Systems. Cambridge University Press, studies in natural language processing edition. find
  25. V. Rieser, O. Lemon, and X. Liu. 2010. Optimising information presentation for spoken dialogue systems. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1009–1018. find
  26. A. Stent, R. Prasad, and M. Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pages 79–86. find
  27. I. Sutskever, O. Vinyals, and Q. VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. arXiv:1409.3215. find
  28. O. Vinyals, Ł. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton. 2015. Grammar as a Foreign Language. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, R. Garnett, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2755–2763. find
  29. M. A. Walker, O. Rambow, and M. Rogati. 2001. SPoT: a trainable sentence planner. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 1–8, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  30. T.-H. Wen, M. Gasic, D. Kim, N. Mrksic, P.-H. Su, D. Vandyke, and S. Young. 2015a. Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page…
  31. T.-H. Wen, M. Gasic, N. Mrkšić, P.-H. Su, D. Vandyke, and S. Young. 2015b. Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, Lisbon, Portugal. Asso…
  32. Y. W. Wong and R. J. Mooney. 2007. Generation by inverting a semantic parser that uses statistical machine translation. In Proceedings of Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT-07), pages 172– 179.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dusek-2016-sequence-sequence-generation-spoken-dialogue/