Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1603.03827

Summary

Most neural short-text classifiers handle each sentence or utterance on its own, even though such texts usually appear in sequences such as dialogs or documents. Lee and Dernoncourt test whether context from earlier utterances helps, using dialog act classification as the task. Their model has two stages. First, an encoder (an LSTM with pooling, or a CNN with max pooling) turns each utterance into a vector. Second, a two-layer feedforward network predicts the dialog act. Its first layer can look at the vectors of up to d1 previous utterances, and its second layer can look at up to d2 previous class representations. The authors evaluate on DSTC 4, MRDA and SwDA, using pretrained word embeddings and no hand-crafted features. Adding context at the utterance-vector level helps most. Adding it at the class level helps less, and combining both levels brings no consistent gain. The resulting system matches or beats earlier feature-based methods on all three corpora. This shows that simple sequential context makes neural dialog act tagging competitive without manual feature engineering.

Key points

Test accuracy (CNN / LSTM):
- DSTC 4: 65.5 / 66.2
- MRDA: 84.6 / 84.3
- SwDA: 73.1 / 69.6
Earlier baselines: SVM 57.0 (DSTC 4), Naive Bayes 82.0 (MRDA), Memory-based Learning 72.3 (SwDA). Human interlabeler agreement on SwDA is 84.0.
Effect of context: Raising d1 while keeping d2 = 0 improved accuracy by 1.3–4.2 percentage points. Across settings, the CNN was generally slightly better than the LSTM, with a clear gap only on SwDA. GRUs performed worse than the LSTM.
Embeddings: 300-dimensional word2vec Google News vectors were used for DSTC 4, and 200-dimensional GloVe Twitter vectors for MRDA and SwDA. These were chosen over SENNA and RNNLM alternatives.
Training: Adadelta optimizer, dropout of 0.5 after pooling, and early stopping with a patience of 10 epochs. The data splits are published at github.com/Franck-Dernoncourt/naacl2016.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

75 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.03827 · arXiv 1603.03827

@misc{lee2016sequential,
  title = {{Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks}},
  author = {Lee, Ji Young and Dernoncourt, Franck},
  year = {2016},
  doi = {10.48550/arxiv.1603.03827},
  eprint = {1603.03827},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.03827}
}

References

3 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

[Ang et al.2005] Jeremy Ang, Yang Liu, and Elizabeth Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. In ICASSP (1), pages 1061–1064. [Blunsom et al.2014] Phil Blunsom, Edward Grefenstette, Nal Kalchbrenner, et al. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. [Cho et al.2014] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. [Collobert et al.2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. [Collobert2011] Ronan Collobert. 2011. Deep learning for efficient discriminative parsing. In International Conference on Artificial Intelligence and Statistics, number EPFL-CONF-192374. [Dernoncourt et al.2016] Franck Dernoncourt, Ji Young Lee, Trung H. Bui, and Hung H. Bui. 2016. Adobe-MIT submission to the DSTC 4 Spoken Language Understanding pilot task. In 7th International Workshop on Spoken Dialogue Systems (IWSDS). [Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [Janin et al.2003] Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The ICSI meeting corpus. In Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03). 2003 IEEE International Conference on, volume 1, pages I–364. IEEE. [Ji and Bilmes2006] Gang Ji and Jeff Bilmes. 2006. Backoff model training using partially observed data: application to dialog act tagging. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 280– 287. Association for Computational Linguistics. [Jurafsky et al.1997] Dan Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual. Institute of Cognitive Science Technical Report, pages 97–102. ↗ our card
[Kalchbrenner et al.2014] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188. [Kim et al.2015] Seokhwan Kim, Luis Fernando D’Haro, Rafael E. Banchs, Jason Williams, and Matthew Henderson. 2015. Dialog State Tracking Challenge 4: Handbook. [Kim et al.2016] Seokhwan Kim, Luis Fernando D’Haro, Rafael E. Banchs, Jason Williams, and Matthew Henderson. 2016. The Fourth Dialog State Tracking Challenge. In Proceedings of the 7th International Workshop on Spoken Dialogue Systems (IWSDS). [Kim2014] Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746–1751. Association for Computational Linguistics. [Lendvai and Geertzen2007] Piroska Lendvai and Jeroen Geertzen. 2007. Token-based chunking of turn-internal dialogue act sequences. In Proceedings of the 8th SIGDIAL Workshop on Discourse and Dialogue, pages 174–181. [Mikolov et al.2011] Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. Rnnlm-recurrent neural network language modeling toolkit. In Proc. of the 2011 ASRU Workshop, pages 196–201. [Mikolov et al.2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. [Mikolov et al.2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [Nakagawa et al.2010] Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 786–794. Association for Computational Linguistics. [Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12:1532–1543. [Reithinger and Klesen1997] Norbert Reithinger and Martin Klesen. 1997. Dialogue act classification using language models. In EuroSpeech. Citeseer. ↗ our card
[Rotaru2002] Mihai Rotaru. 2002. Dialog act tagging using memory-based learning. Term project, University of Pittsburgh, pages 255–276. [Schuster and Paliwal1997] Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. [Shriberg et al.2004] Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. Technical report, DTIC Document. [Silva et al.2011] Joao Silva, Luı́sa Coheur, Ana Cristina Mendes, and Andreas Wichert. 2011. From symbolic to sub-symbolic information in question classification. Artificial Intelligence Review, 35(2):137–154. [Socher et al.2012] Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. Association for Computational Linguistics. [Stolcke et al.2000] Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. [Wang and Manning2012] Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 90–94. Association for Computational Linguistics. [Zeiler2012] Matthew D Zeiler. 2012. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701. ↗ our card