Elamur Research⌕
☰
home / papers / 2018 / Semantic Parsing for Task Oriented Dialog using Hierarchical Representations

Semantic Parsing for Task Oriented Dialog using Hierarchical Representations

Summary

Conventional task-oriented assistants map each request to a single intent and assign each token one slot tag. That flat scheme cannot handle nested requests such as asking for directions to an event, where finding the event is itself a sub-task. Logical forms can express such requests, but they are expensive to annotate and hard to parse. The authors introduce TOP (Task Oriented Parsing), a tree-shaped middle ground between these two options. Words are the leaves, internal nodes are intents or slots, and a slot may contain a nested intent. Because the trees look like constituency parses, annotators only need to mark spans, and existing phrase-structure parsers can be reused without much change. The authors crowdsourced roughly 44k navigation and event requests, labeled them with TOP, and released the dataset. They then compared a discriminatively trained RNNG parser with CNN, LSTM and Transformer seq2seq models built in fairseq. RNNG gave the best results on every metric, with 78.51% exact-match accuracy, and its inference runs in linear time. The study shows that a modest increase in structure covers far more real requests while keeping annotation and parsing practical.

Key points

  • Coverage: About 30% of the crowdsourced queries need compositional structure that flat intent-slot tagging cannot express. Only around 0.3% would need a graph rather than a tree.
  • Dataset: The released TOP corpus has 44,783 utterances with 25 intents and 36 slots.
    • The split is 31,279 for training, 4,462 for validation and 9,042 for testing.
    • 35% of the trees are deeper than 2, and 4,646 utterances combine navigation and event intents.
  • Annotation agreement: 63.40% of items were settled by the first two annotators, and 94.09% once a third annotator was added. Agreement was similar for flat and nested trees.
  • Model comparison:
    • RNNG (greedy decoding) reached 78.51 exact match, 90.23 labeled-bracketing F1 and 84.27 Tree-Labeled F1, with 100% valid trees.
    • The best seq2seq model, a CNN with a Limited Output Token Vocabulary (LOTV), reached 75.87 exact match.
    • The Transformer scored 72.20 exact match.
  • Further RNNG results:
    • Top-5 accuracy for RNNG rises to 92.48.
    • In ablations, removing the buffer LSTM collapses exact match to 13.78, which shows that the model's structural bias is central to its performance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 96 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1300

@inproceedings{gupta2018semantic,
  title = {{Semantic Parsing for Task Oriented Dialog using Hierarchical Representations}},
  author = {Gupta, Sonal and Shah, Rushin and Mohit, Mrinal and Kumar, Anuj and Lewis, Mike},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1300},
  url = {https://doi.org/10.18653/v1/d18-1300}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015. Broad-coverage ccg semantic parsing with AMR. find
  2. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1699–1710. representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou… find our card
  4. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  5. Ezra W. Black, Steven Abney, Daniel P. Flickinger, Claudia Gdaniec, Ralph Grishman, Philip Harrison, Donald Hindle, Robert J. P. Ingria, Frederick Jelinek, Judith L. Klavans, Mark Y. Liberman, Mitchell P. Marcus, Salim Roukos, Beatrice Santorini, and Tomek Strzalkowski. 1991. A procedure for quantit… find
  6. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proc. of NAACL. find
  7. Xing Fan, Emilio Monti, Lambert Mathias, and Markus Dreyer. 2017. Transfer learning for neural semantic parsing. In Proceedings of the 2nd Workshop on Representation Learning for NLP, ACL. find
  8. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. ArXiv e-prints. find our card
  9. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, pages 2773–2781, Cambridge, MA, USA. MIT Press. find
  10. Jason Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse, 7(3):4–33. find
  11. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In EMNLP. find
  12. Luke S Zettlemoyer and Michael Collins. 2012. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv:1207.1420. find
  13. S. Zhu and K. Yu. 2017. Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5675–5679. find
  14. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. find
  15. Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic ccg grammars from logical form with higher-order unification. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 1223–1233. Association for Computational… find
  16. Percy Liang. 2016. Learning executable semantic parsers for natural language understanding. Commun. ACM, 59(9):68–76. find
  17. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. In INTERSPEECH. find
  18. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. 2013. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In Interspeech. find
  19. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gupta-2018-semantic-parsing-task-oriented-dialog/