Elamur Research
☰
home / papers / 2017 / Regularizing Neural Networks by Penalizing Confident Output Distributions

Regularizing Neural Networks by Penalizing Confident Output Distributions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1701.06548

Summary

Common regularizers such as dropout, weight decay and batch normalization act on a network's weights or hidden activations. This paper instead regularizes the output distribution. The authors add a confidence penalty to the training objective: the standard negative log-likelihood minus β times the entropy of the softmax prediction. This discourages the overly peaked outputs that often signal overfitting. They also describe annealed and hinge-thresholded variants. In addition, they show that uniform label smoothing differs from this penalty only in the direction of the KL divergence to the uniform distribution. Both regularizers, including a unigram form of label smoothing, are evaluated on six benchmarks: MNIST, CIFAR-10, Penn Treebank, WMT'14 English-to-German, TIMIT and WSJ, using TensorFlow implementations. When added to strong existing models without retuning their hyperparameters, both methods improved results across tasks. The practical value is a cheap loss term with a single hyperparameter. It does not depend on how the network is parameterized and transfers across vision, language and speech.

Key points

  • Objective: loss = −log p(y|x) − β·H(p). The entropy gradient with respect to each logit has a simple closed form: each class probability multiplies how far that class's negative log-probability lies from the entropy. A thresholded version penalizes only outputs whose entropy falls below Γ. It converges faster but adds a hyperparameter, so most experiments use the plain β version.
  • Link to label smoothing: uniform label smoothing corresponds to adding D_KL(u‖p). Swapping the arguments to D_KL(p‖u) gives the confidence penalty, which suggests other target distributions could be used.
  • MNIST (permutation-invariant, 2 hidden layers × 1024 ReLU units): test error was 1.17 ± 0.06% with the confidence penalty, 1.23% with label smoothing and 1.28% with dropout. The best settings were β = 1.0 and a smoothing value of 0.1.
  • Training behaviour: both output regularizers produced smoother softmax distributions than dropout, which pushes probabilities toward 0 or 1. They also yielded smaller gradient norms and converged faster.
  • Excluded baseline: virtual adversarial training (VAT) was not compared because its multiple hyperparameters and extra forward/backward passes make it much more expensive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

156 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 149 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1701.06548 · arXiv 1701.06548

@misc{pereyra2017regularizing,
  title = {{Regularizing Neural Networks by Penalizing Confident Output Distributions}},
  author = {Pereyra, Gabriel and Tucker, George and Chorowski, Jan and Kaiser, Łukasz and Hinton, Geoffrey},
  year = {2017},
  doi = {10.48550/arxiv.1701.06548},
  eprint = {1701.06548},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1701.06548}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016. find our card
  2. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4945–4949, March 2016. doi: 10.1109/ICASSP.2016.7472618. — [doi:10.1109/icassp.2016.7472618](https://doi.org/10.1109/icassp.2016.7472618) find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  4. Adam L Berger, Vincent J Della Pietra, and Stephen A Della Pietra. A maximum entropy approach to natural language processing. Computational linguistics, 22(1):39–71, 1996. find our card
  5. Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535–541. ACM, 2006. find
  6. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. N-gram counts and language models from the common crawl. In LREC, volume 2, pp. 4. Citeseer, 2014. find
  7. Parameters Validation Test 26.5M 5.7M 5.1M 5.9M -- - - 27.3 18.6 14.7 12.9 6.6M 6.6M 6.6M 6.6M 17.9 14.7 14.0 ± 0.25 17.2 14.2 11.3 11.0 ± 0.35 12.7 William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv:1508.01211, 2015. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) find
  8. William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. Latent sequence decompositions. arXiv preprint arXiv:1610.03035, 2016. — [arXiv:1610.03035](https://arxiv.org/abs/1610.03035) find
  9. Sébastien Jean Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. 2015. find our card
  10. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pp. 577–585, 2015. find our card
  11. Yarin Gal. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287, 2015. find our card
  12. Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, volume 14, pp. 1764–1772, 2014. find
  13. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369–376. ACM, 2006. find
  14. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645–6649. find
  15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015. find
  16. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. find our card
  17. Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016a. — [arXiv:1608.06993](https://arxiv.org/abs/1608.06993) find
  18. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382, 2016b. find
  19. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. find
  20. Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957. find
  21. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  22. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648, 2016. — [arXiv:1605.07648](https://arxiv.org/abs/1605.07648) find
  23. Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, and Ilya Sutskever. Learning online alignments with continuous rewards policy gradient. arXiv preprint arXiv:1608.01281, 2016. — [arXiv:1608.01281](https://arxiv.org/abs/1608.01281) find
  24. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015. find
  25. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993. find our card
  26. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. find our card
  27. David Miller, Ajit V Rao, Kenneth Rose, and Allen Gersho. A global optimization technique for statistical classifier design. IEEE Transactions on Signal Processing, 44(12):3108–3122, 1996. find
  28. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing by virtual adversarial examples. arXiv preprint arXiv:1507.00677, 2015. — [arXiv:1507.00677](https://arxiv.org/abs/1507.00677) find
  29. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783, 2016. find
  30. Abdel-rahman Mohamed, George E Dahl, and Geoffrey Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):14–22, 2012. find
  31. Mohammad Norouzi, Dale Schuurmans, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, and Yonghui Wu. Reward augmented maximum likelihood for neural structured prediction. In Advances in Neural Information Processing Systems, 2016. find
  32. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. The kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Unders… find
  33. IEEE Signal Processing Society, December 2011. IEEE Catalog No.: CFP11SRW-USB. find
  34. Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016. find our card
  35. Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014. — [arXiv:1412.6596](https://arxiv.org/abs/1412.6596) find
  36. Kenneth Rose. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEEE, 86(11):2210–2239, 1998. find
  37. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. find
  38. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9, 2015a. find
  39. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015b. find
  40. László Tóth. Combining time-and frequency-domain convolution in convolutional neural network-based phone recognition. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 190–194. find
  41. Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1058–1066, 2013. find
  42. Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991. find
  43. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  44. Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, and Qi Tian. Disturblabel: Regularizing cnn on the loss layer. arXiv preprint arXiv:1605.00055, 2016. — [arXiv:1605.00055](https://arxiv.org/abs/1605.00055) find
  45. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. find our card
  46. Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016. find
  47. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnı́k, and Jürgen Schmidhuber. Recurrent highway networks. arXiv preprint arXiv:1607.03474, 2016. find