Elamur Research⌕
☰
home / papers / 2014 / One weird trick for parallelizing convolutional neural networks

One weird trick for parallelizing convolutional neural networks

Paper · 2014 · Alex Krizhevsky

Summary

Training large convolutional networks on several GPUs is limited by communication. Pure data parallelism must exchange many parameters, and pure model parallelism must exchange many activations. Krizhevsky notes that the two layer types behave very differently. Convolutional layers do most of the computation but hold few weights. Fully-connected layers hold most of the weights but little computation.

He therefore proposes a hybrid scheme:

  • Convolutional layers use data parallelism: each GPU processes its own batch.
  • Fully-connected layers use model parallelism: the last convolutional activities are shared across GPUs.

He describes three ways to exchange these activities, some of which overlap communication with computation. He also describes a variant that updates the fully-connected weights more often. This variant uses a smaller effective batch in those layers and no longer exactly reproduces synchronous SGD.

The experiments use a single-tower version of the ILSVRC 2012 winning network, trained on ImageNet (ILSVRC 2012). On 8 GPUs, training runs about 6.2 times faster with only a small loss in accuracy. The work offers a simple, practical recipe for scaling convnet training across GPUs without redesigning the architecture.

Key points

  • Layer profile motivating the method:
    • Convolutional layers: about 90–95% of computation and about 5% of parameters.
    • Fully-connected layers: about 5–10% of computation and about 95% of parameters.
  • Speedups over 1 GPU (98.05 h): 1.95x on 2 GPUs, 3.74x on 4 GPUs and 6.25x on 8 GPUs (15.68 h).
  • Top-1 validation error rose from 42.33% on 1 GPU to 43.28% on 8 GPUs with batch 1024. Keeping the fully-connected batch at 128 reduced the 8-GPU error to 42.86%.
  • In practice, the best learning-rate heuristic was to scale the rate linearly with batch size. This worked better than the square-root scaling suggested by the author's analysis.
  • The model uses 1000 independent logistic output units instead of a softmax, which makes parallelization easier. Every run was trained for 90 epochs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 works

Cite

Cite as Alex Krizhevsky. 2014. One weird trick for parallelizing convolutional neural networks. arXiv:1404.5997.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1404.5997 · arXiv 1404.5997

@misc{krizhevsky2014one,
  title = {{One weird trick for parallelizing convolutional neural networks}},
  author = {Alex Krizhevsky},
  year = {2014},
  doi = {10.48550/arxiv.1404.5997},
  eprint = {1404.5997},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1404.5997}
}

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/