Elamur Research
☰
home / papers / 2015 / Highway Networks

Highway Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1505.00387

Summary

Adding layers usually makes neural networks more expressive, but plain deep feedforward models become hard to train with gradient descent unless their initialization is carefully tuned. This ICML 2015 Deep Learning Workshop paper introduces highway networks. In each layer, a learned sigmoid transform gate T decides how much of the output comes from the usual nonlinear transform H and how much is the input carried through unchanged (weight 1 − T). The idea borrows from LSTM gating. Setting the gate biases to negative values at the start makes every layer begin close to an identity mapping, whatever activation H uses.

On MNIST, highway networks with up to 100 layers trained with SGD and momentum showed no sign of optimization getting harder as depth grew. Plain networks with a similar parameter count clearly degraded with depth. On CIFAR-10, highway versions of Fitnets matched or beat them without the two-stage, teacher-based training that Fitnets require. This matters because it allows experiments with very deep models and with activations that lack reliable initialization schemes.

Key points

  • Layer formula: y = H(x)·T(x) + x·(1 − T(x)), with T(x) = σ(W_T x + b_T). Changes in dimensionality are handled by plain layers inserted between highway stacks. Convolutional highway layers use zero-padding.
  • MNIST optimization test: one plain first layer, then 9, 19, 49 or 99 plain or highway layers, then softmax. Highway layers had 50 units and plain layers 71, to keep parameter counts comparable. Each setup received 40 random-search runs, with gate bias searched between −1 and −10. The 100-layer highway net reached roughly 10× lower training error than the 10-layer highway net and converged faster than the plain nets. A 900-layer highway net on CIFAR-100 was still at 80 epochs at the time of writing and showed no difficulties so far.
  • CIFAR-10 vs Fitnets:
    • Highway 1 (11 layers, ~236K params) reached 89.18%, against 89.01% for Fitnet 1.
    • Highway 2 (19 layers, ~2.3M params) reached 92.24%, against 91.61% for Fitnet 4.
    • Thinner 19-layer (~1.4M params) and 32-layer (~1.25M params) highway nets scored about 90.7% and 90.3%. This is close to the 90.18% of the ~9M-parameter teacher.
  • Gate analysis (50-layer nets, MNIST and CIFAR-100):
    • Gate biases initialized at −2 and −4 mostly became even more negative during training.
    • Gate activity was sparse and depended on the input.
    • Most block outputs stayed constant across layers, so the representation changed mainly in roughly the first 10 layers (MNIST) and 30 layers (CIFAR-100).
  • An extended full version of the study is available as arXiv:1507.06228.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

150 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 144 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1505.00387 · arXiv 1505.00387

@misc{srivastava2015highway,
  title = {{Highway Networks}},
  author = {Srivastava, Rupesh Kumar and Greff, Klaus and Schmidhuber, Jürgen},
  year = {2015},
  doi = {10.48550/arxiv.1505.00387},
  eprint = {1505.00387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1505.00387}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1798–1828, 2013. URL http://ieeexplore.ieee.org/ xpls/abs\_all.jsp?arnumber=6472238. ↗ our card
  2. Ciresan, Dan, Meier, Ueli, Masci, Jonathan, and Schmidhuber, Jürgen. A committee of neural networks for traffic sign classification. In Neural Networks (IJCNN), The 2011 International Joint Conference on, pp. 1918–1921. find
  3. IEEE, 2011a. URL http://ieeexplore.ieee. org/xpls/abs\_all.jsp?arnumber=6033458. ↗
  4. Ciresan, Dan, Meier, Ueli, and Schmidhuber, Jürgen. Multi-column deep neural networks for image classification. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. find
  5. Ciresan, DC, Meier, Ueli, Masci, Jonathan, Gambardella, Luca M, and Schmidhuber, Jürgen. Flexible, high performance convolutional neural networks for image classification. In IJCAI, 2011b. URL http://www. aaai.org/ocs/index.php/IJCAI/IJCAI11/ paper/download/3098/3425%0020http: //dl.acm.org/citation… find
  6. Gers, Felix A., Schmidhuber, Jürgen, and Cummins, Fred. Learning to forget: Continual prediction with LSTM. In ICANN, volume 2, pp. 850–855, 1999. URL http://ieeexplore.ieee.org/ xpls/abs\_all.jsp?arnumber=818041. ↗
  7. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 249–256, 2010. URL http: //machinelearning.wustl.edu/mlpapers/ paper\_files/AISTATS2010\_GlorotB10.pdf. find
  8. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 [cs], September 2014. URL http: //arxiv.org/abs/1409.1556. find our card
  9. Håstad, Johan. Computational limitations of small-depth circuits. MIT press, 1987. URL http://dl.acm. org/citation.cfm?id=SERIES9056.27031. ↗
  10. Srivastava, Rupesh Kumar, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, Jürgen. Understanding locally competitive networks. In International Conference on Learning Representations, 2015. URL http: //arxiv.org/abs/1410.1165. — [arXiv:1410.1165](https://arxiv.org/abs/1410.1165) find
  11. Håstad, Johan and Goldmann, Mikael. On the power of small-depth threshold circuits. Computational Complexity, 1(2):113–129, 1991. URL http://link.springer.com/article/10. 1007/BF01272517. ↗
  12. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. arXiv:1502.01852 [cs], February 2015. URL http: //arxiv.org/abs/1502.01852. find
  13. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short term memory. Technical Report FKI-207-95, Technische Universität München, München, August 1995. URL http://citeseerx.ist.psu.edu/ viewdoc/summary?doi=10.1.1.51.3117. ↗
  14. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012. URL http://books.nips.cc/ papers/files/nips25/NIPS2012\_0534.pdf. ↗
  15. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. pp. 562–570, 2015. URL http://jmlr.org/ proceedings/papers/v38/lee15a.html. ↗
  16. Montufar, Guido F, Pascanu, Razvan, Cho, Kyunghyun, and Bengio, Yoshua. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems. 2014. URL http://papers.nips.cc/paper/5422-on-the-number-of-linear-regions-of-deep-neural-networks.pdf. ↗
  17. Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua. FitNets: Hints for thin deep nets. arXiv:1412.6550 [cs], December 2014. URL http://arxiv.org/abs/1412.6550. ↗
  18. Saxe, Andrew M., McClelland, James L., and Ganguli, Surya. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120 [cond-mat, q-bio, stat], December 2013. URL http://arxiv.org/abs/1312. 6120. ↗
  19. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. arXiv:1409.4842 [cs], September 2014. URL http://arxiv.org/abs/ 1409.4842. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) ↗