Elamur Research
☰
home / papers / 2013 / Recurrent Convolutional Neural Networks for Discourse Compositionality

Recurrent Convolutional Neural Networks for Discourse Compositionality

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1306.3584

Summary

The authors argue that meaning composes at two levels: words build sentences, and sentences build discourse such as dialogues. They propose a separate neural model for each level. The sentence model is a hierarchical convolutional neural network (HCNN). It applies one-dimensional convolutions to each feature dimension of the word vectors, with kernels that grow larger at each layer, and uses sigmoid activations. This makes it sensitive to word order and able to handle sentences of any length. The discourse model, a recurrent convolutional network (RCNN), builds on the sentence model. Its recurrent state is conditioned on both the current sentence and the identity of the speaker, so it tracks the order of utterances and the interaction between participants. The models were tested on dialogue act tagging in the Switchboard Dialogue Act (SwDA) corpus. They used no feature engineering, no pretraining and only greedy decoding, yet beat the earlier LM-HMM approach. This matters because it shows that learned compositional representations can capture pragmatic structure across utterances.

Key points

  • Dataset: SwDA, which tags utterances with 42 dialogue act categories, Statement and Opinion among them. The authors used the split from Stolcke et al. (2000): 1115 training and 19 test dialogues. Only transcripts, act labels and speaker labels were used.
  • Training setup:
    • Word vectors of size 25, randomly initialised.
    • Cross-entropy loss with l2 regularisation, optimised by L-BFGS in mini-batches.
    • Recurrence truncated to depth 2; deeper truncation gave no improvement.
  • Results: The RCNN reached 73.9% accuracy, compared with:
    • LM-HMM trigram: 71.0%
    • LM-HMM bigram: 70.6%
    • LM-HMM unigram: 68.2%
    • Majority baseline: 31.5%
    • Random baseline: 2.4%
    • Inter-annotator agreement (the practical ceiling): 84%
  • Weaker assumption than the baseline: The RCNN predicts each act using only the utterances seen so far. LM-HMM needs the entire dialogue.
  • Learned representations: Nearest neighbours of the hidden-state dialogue vectors group dialogues by pragmatic role rather than by topic. For example, a dialogue about repairing a car sits close to one about managing money.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Speech act theory

Cited by

75 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1306.3584 · arXiv 1306.3584

@misc{kalchbrenner2013recurrent,
  title = {{Recurrent Convolutional Neural Networks for Discourse Compositionality}},
  author = {Kalchbrenner, Nal and Blunsom, Phil},
  year = {2013},
  doi = {10.48550/arxiv.1306.3584},
  eprint = {1306.3584},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1306.3584}
}

References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John L. Austin. 1962. How to do things with words. Oxford: Clarendon. find
  2. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In EMNLP, pages 1183–1193. ACL. find our card
  3. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In EMNLP-CoNLL, pages 546–556. ACL. find
  4. Sasha Calhoun, Jean Carletta, Jason M. Brenier, Neil Mayo, Dan Jurafsky, Mark Steedman, and David Beaver. 2010. The nxt-format switchboard corpus: a rich resource for investigating the syntax, semantics, pragmatics and prosody of dialogue. Language Resources and Evaluation, 44(4):387–419. find
  5. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning, ICML. find
  6. Gottlob Frege. 1892. Über Sinn und Bedeutung. Zeitschrift für Philosophie und philosophische Kritik, 100. find
  7. Edward Grefenstette, Mehrnoosh Sadrzadeh, Stephen Clark, Bob Coecke, and Stephen Pulman. 2011. Concrete sentence spaces for compositional distributional models of meaning. CoRR, abs/1101.0309. ↗
  8. Karl Moritz Hermann and Phil Blunsom. 2013. The Role of Syntax in Vector Space Models of Compositional Semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria, August. Association for Computational Linguistics. Forthcoming. find
  9. Rosie Jones, Benjamin Rey, Omid Madani, and Wiley Greiner. 2006. Generating query substitutions. In Les Carr, David De Roure, Arun Iyengar, Carole A. Goble, and Michael Dahlin, editors, WWW, pages 387–396. ACM. find
  10. Kepa Korta and John Perry. 2012. Pragmatics. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Winter 2012 edition. find
  11. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 2001. Gradient-based learning applied to document recognition. In Intelligent Signal Processing, pages 306–351. IEEE Press. find
  12. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Takao Kobayashi, Keikichi Hirose, and Satoshi Nakamura, editors, INTERSPEECH, pages 1045–1048. ISCA. find
  13. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  14. Christopher Potts. 2011. Pragmatics. In Ruslan Mitkov, editor, The Oxford Handbook of Computational Linguistics. Oxford University Press, 2 edition. find
  15. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning internal representations by error propagation. MIT Press Computational Models Of Cognition And Perception Series, page 318362. find
  16. Christian Scheible and Hinrich Schütze. 2013. Cutting recursive autoencoder trees. CoRR, abs/1301.2811. ↗
  17. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  18. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca A. Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialog act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–373. find
  19. Ilya Sutskever, James Martens, and Geoffrey E. Hinton. 2011. Generating text with recurrent neural networks. In Lise Getoor and Tobias Scheffer, editors, ICML, pages 1017–1024. Omnipress. find
  20. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. J. Artif. Intell. Res. (JAIR), 37:141–188. find
  21. Sida Wang and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL (2), pages 90–94. The Association for Computer Linguistics. find
  22. Jason D. Williams. 2012. A belief tracking challenge task for spoken dialog systems. In NAACL-HLT Workshop on Future Directions and Needs in the Spoken Dialog Community: Tools and Data, SDCTD ’12, pages 23–24, Stroudsburg, PA, USA. Association for Computational Linguistics. find