Elamur Research⌕
☰
home / papers / 2017 / A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification

A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification

Summary

One-layer CNNs, such as Kim's (2014) model, are strong and fast baselines for sentence classification. They still require many architectural and hyperparameter choices, and practitioners have had little evidence about which of those choices actually matter. Zhang and Wallace address this gap with a systematic sensitivity analysis. They fix a baseline configuration, change one component at a time (input embeddings, filter region size, number of feature maps, activation, pooling, dropout, l2 norm constraint) and measure the effect on nine sentence classification datasets. Because CNN training is highly stochastic, each setting is evaluated with ten repetitions of 10-fold cross-validation, and the authors report both means and ranges. The analysis shows that some settings deserve tuning, chiefly filter region size, feature map count and embeddings. Others, such as pooling, have a choice that wins across datasets, and heavy regularization rarely helps. The resulting practical guide gives sensible defaults and search ranges, which are also useful as starting bounds for random search or Bayesian optimization.

Key points

  • Datasets: nine in total: Irony (16,006 reddit sentences), Opi (Opinosis, 51 topics of roughly 100 sentences each), plus CR, TREC, Subj, MPQA, MR and the two Stanford Sentiment Treebank variants, SST-1 and SST-2.
    • Irony was balanced by under-sampling negatives and is scored with AUC instead of accuracy.
    • SST models were trained and tested on sentences only, not phrases.
  • Baseline configuration:
    • Google word2vec embeddings (300-dimensional)
    • region sizes (3,4,5) with 100 feature maps each
    • ReLU activation
    • 1-max pooling
    • dropout 0.5
    • l2 norm constraint of 3
  • Region size and pooling: region size can strongly affect results; a line search over about 1–10 is suggested. 1-max pooling outperformed the other pooling strategies on every dataset.
  • Regularization: dropout gave little benefit, both at the softmax layer and on the convolution layer, and high rates hurt badly. The l2 constraint mostly did not help (except on Opi) and harmed CR.
  • Embeddings: word2vec and GloVe (840B-token web corpus) each win on different tasks. Both beat one-hot inputs.

Interesting ideas

  • I1. Repeat cross-validation and report ranges: rerunning the whole CV procedure several times exposes how much a score moves due to random initialization, dropout and SGD alone. This applies to any stochastic model and prevents over-reading small differences between configurations.
  • I2. One-factor-at-a-time sweeps from a fixed baseline: this cheaply separates the knobs worth tuning from those that barely matter. It is a useful first step before setting up search ranges for a more expensive optimizer.
  • I3. Find the best single filter size, then combine its neighbours: sets of several sizes close to the best single size consistently beat that size alone. The trick generalizes to other n-gram-window style models.
  • I4. Watch the edge of the search range: if the best number of feature maps lands near the upper limit (e.g., 600), extend the range. When enlarging capacity starts to hurt, raise dropout. This is a simple diagnostic for any capacity hyperparameter.
  • I5. Balance a skewed dataset by under-sampling and evaluate with AUC: the authors found under-sampling worked better than over-sampling for the imbalanced Irony data. This is relevant to rare-class detection tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 97 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhang2017sensitivity,
  title = {{A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification}},
  author = {Zhang, Ye and Wallace, Byron},
  booktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2017},
  url = {https://aclanthology.org/I17-1026/}
}

References

41 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio. 2009. Learning deep architectures for ai. Foundations and trends in Machine Learning, 2(1):1–127. find
  2. Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: a graph-based approach to abstractive summarization of highly redundant opinions. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 340–348. Association for Computational Linguistics. find our card
  3. Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437– 478. Springer. find
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155. find
  5. Yoav Goldberg. 2015. A primer on neural network models for natural language processing. arXiv preprint arXiv:1510.00726. find
  6. James Bergstra, Daniel Yamins, and David Daniel Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. find
  7. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
  8. Y-Lan Boureau, Francis Bach, Yann LeCun, and Jean Ponce. 2010a. Learning mid-level features for recognition. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2559–2566. IEEE. find
  9. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. ACM. find
  10. Y-Lan Boureau, Jean Ponce, and Yann LeCun. 2010b. A theoretical analysis of feature pooling in visual recognition. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 111–118. find
  11. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. find our card
  12. Y-Lan Boureau, Nicolas Le Roux, Francis Bach, Jean Ponce, and Yann LeCun. 2011. Ask the locals: multi-way local pooling for image recognition. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2651–2658. IEEE. find
  13. Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. Springer. find our card
  14. Rie Johnson and Tong Zhang. 2014. Effective use of word order for text categorization with convolutional neural networks. arXiv preprint arXiv:1412.1058. find our card
  15. Thomas M Breuel. 2015. The effects of hyperparameters on sgd training of neural networks. arXiv preprint arXiv:1508.02788. — [arXiv:1508.02788](https://arxiv.org/abs/1508.02788) find
  16. Rie Johnson and Tong Zhang. 2015. Semi-supervised convolutional neural networks for text categorization via region embedding. In Advances in neural information processing systems, pages 919–927. find
  17. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), volume 1, pages 740–750. find our card
  18. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 655–665, Baltimore, Maryland. Association for Computational… find our card
  19. Adam Coates, Andrew Y Ng, and Honglak Lee. 2011. An analysis of single-layer networks in unsupervised feature learning. In International conference on artificial intelligence and statistics, pages 215– 223. find
  20. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  21. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. find our card
  22. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. find
  23. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  24. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521(7553):436–444. find
  25. Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. 2001. Incorporating second-order functional knowledge for better option pricing. Advances in Neural Information Processing Systems, pages 472–478. find
  26. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1–7. Association for Computational Linguistics. find our card
  27. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165–210. find
  28. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. ICML, volume 30. find
  29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  30. Dani Yogatama and Noah A Smith. 2015. Bayesian optimization of text representations. arXiv preprint arXiv:1503.00693. find
  31. Ye Zhang, Matthew Lease, and Byron C Wallace. 2017. Exploiting domain knowledge via grouped weight sharing with application to text categorization. arXiv preprint arXiv:1702.02535. find
  32. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL. find our card
  33. Ye Zhang, Stephen Roller, and Byron Wallace. 2016. Mgnc-cnn: A simple approach to exploiting multiple word embeddings for sentence classification. arXiv preprint arXiv:1603.00968. find
  34. Wenzhe Pei, Tao Ge, and Baobao Chang. 2015. An effective neural network model for graph-based dependency parsing. In Proc. of ACL. find
  35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12:1532–1543. find
  36. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1988. Learning representations by backpropagating errors. Cognitive modeling, 5:3. find
  37. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volum… find
  38. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  39. Byron C Wallace, Laura Kertz Do Kook Choe, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 512–516. find
  40. Byron C Wallace, Kevin Small, Carla E Brodley, and Thomas A Trikalinos. 2011. Class imbalance, redux. In Data Mining (ICDM), 2011 IEEE 11th International Conference on, pages 754–763. IEEE. find
  41. Peng Wang, Jiaming Xu, Bo Xu, Chenglin Liu, Heng Zhang, Fangyuan Wang, and Hongwei Hao. 2015. Semantic clustering and convolutional neural network for short text categorization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2017-sensitivity-analysis-practitioners-guide-convolutional/