Elamur Research⌕
☰
home / papers / 2017 / Very Deep Convolutional Networks for Text Classification

Very Deep Convolutional Networks for Text Classification

Summary

The paper asks whether text classification benefits from the very deep convolutional architectures that transformed computer vision. At the time, NLP models (LSTMs and ConvNets alike) were shallow, and earlier character-level ConvNets stopped at about six convolutional layers. The authors introduce VDCNN, a network that reads raw characters (inputs fixed at 1024 characters) and stacks many small convolutions and pooling steps, following the design rules of VGG and ResNet. They test depths of 9, 17, 29 and 49 layers on the eight large public classification datasets from Zhang et al. (2015), covering sentiment, topic, ontology and question-answer categorisation. Accuracy improves steadily up to 29 layers. VDCNN beats earlier ConvNets on every dataset and sets new state-of-the-art results on most of them. The exceptions are the two smallest sets, AG's news and Sogou news, where n-gram TF-IDF baselines stay ahead. At 49 layers, accuracy drops unless residual shortcuts are added. The work matters because it is the first demonstration that depth pays off for convolutional models in NLP.

Key points

  • Architecture:
    • A 16-dimensional character lookup table feeds one convolution layer with 64 filters of width 3.
    • Convolutional blocks follow with 64, 128, 256 and 512 feature maps, separated by 3 pooling steps that each halve the temporal length.
    • k-max pooling with k=8 then produces a fixed-size output.
    • Three fully connected layers (4096→2048→2048→classes) perform the classification.
  • Benchmarks: evaluation uses the eight datasets of Zhang et al. (2015), including Yelp Review Polarity/Full, Amazon, Yahoo Answers, DBPedia, AG's news and Sogou news.
  • Comparison with other models:
    • On Yelp Review Polarity and DBPedia, the deepest models match n-gram and n-gram TF-IDF baselines.
    • Yang et al.'s hierarchical attention network still wins on Yahoo Answers, which has multi-sentence samples.
  • Effect of depth and shortcuts (Yelp Full test error for depths 9/17/29/49):
    • Without shortcuts: 37.63 / 36.10 / 35.28 / 37.41.
    • With shortcuts: 40.27 / 39.18 / 36.01 / 36.15.
  • Optimization problem at 49 layers: without shortcuts, going from 29 to 49 layers also raises training error from 29.57 to 35.54. This points to an optimization difficulty rather than overfitting.

Interesting ideas

  • I1. Halve length, double channels. Every time the sequence length is halved, the number of feature maps doubles, so each layer costs roughly the same compute and memory. This rule is simple to reuse when designing any deep 1D encoder over long sequences.
  • I2. Raw characters plus tiny kernels. Deep stacks of width-3 convolutions over raw characters can build up larger units (n-grams, stems, words) without a tokenizer. This could suit morphologically rich languages or noisy text, where word-level vocabularies break down.
  • I3. Global k-max pooling only at the top. k-max pooling keeps the strongest features from anywhere in the text while preserving their order. The authors found that max operations at intermediate layers tended to hurt, except on the smallest datasets, which is a useful heuristic for where to place global pooling.
  • I4. Compare training and test error across depths. Tracking both errors at increasing depth, with and without shortcuts, separates optimization failure from overfitting. This cheap diagnostic applies to any "should I go deeper?" experiment.
  • I5. Number of classes as a factor for depth. The authors suggest that tasks with many classes send more gradient signal back through the network and might gain more from depth. This makes the number of labels a factor worth considering when choosing model size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{conneau2017very,
  title = {{Very Deep Convolutional Networks for Text Classification}},
  author = {Conneau, Alexis and Schwenk, Holger and Barrault, Loïc and Lecun, Yann},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  year = {2017},
  doi = {10.18653/v1/e17-1104},
  url = {https://doi.org/10.18653/v1/e17-1104}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Rejean Ducharme, and Pascal Vincent. 2001. A neural probabilistic language model. In NIPS, volume 13, pages 932–938, Vancouver, British Columbia, Canada. find
  2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155. find
  3. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, pages 160–167, Helsinki, Finland. find
  4. Ronan Collobert, Jason Weston Lon Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, pages 2493–2537. find our card
  5. Cı́cero Nogueira Dos Santos and Maira Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In COLING, pages 69–78, Dublin, Ireland. find our card
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, Santiago, Chile. find
  7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, Las Vegas, Nevada, USA. find
  8. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630–645, Amsterdam, Netherlands. Springer. find
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  10. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448–456, Lille, France. find
  11. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 655–665, Baltimore, Maryland, USA. find our card
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. find our card
  13. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, Lake Tahoe, California, USA. find
  14. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. find
  15. David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110. find
  16. Pedro HO Pinheiro and Ronan Collobert. 2014. Recurrent convolutional neural networks for scene labeling. In ICML, pages 82–90, Beijing, China. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. pages 1715–1725. find
  18. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR, San Diego, California, USA. find our card
  19. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the conference on empirical methods in natural language processing, pages 151–161, Edinburgh, UK. Associati… find our card
  20. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In Interspeech, pages 194–197, Portland, Oregon, USA. find
  21. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112, Montreal, Canada. find
  22. Yijun Xiao and Kyunghyun Cho. 2016. Efficient character-level document classification by combining convolution and recurrent layers. find
  23. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL-HLT, pages 1480–1489, San Diego, California, USA. find our card
  24. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818– 833, Zurich, Switzerland. Springer. find our card
  25. Xiang Zhang and Yann LeCun. 2015. Text understanding from scratch. arXiv preprint arXiv:1502.01710. — [arXiv:1502.01710](https://arxiv.org/abs/1502.01710) find
  26. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS, pages 649–657, Montreal, Canada. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/