Elamur Research⌕
☰
home / papers / 2016 / Feature Pyramid Networks for Object Detection

Feature Pyramid Networks for Object Detection

Paper · 2016 · Tsung-Yi Lin, Piotr Dollar, Ross Girshick et al.

Summary

Detecting objects whose size varies widely is a core difficulty in computer vision. The classic remedy is to run a feature extractor on every level of an image pyramid. This gives semantically strong features at all resolutions, but it multiplies inference time and needs too much memory for end-to-end training, so detectors such as Fast/Faster R-CNN usually work at a single scale.

The authors propose the Feature Pyramid Network (FPN), which reuses the multi-resolution feature maps a ConvNet (here ResNet) already computes. A top-down path upsamples coarse but semantically rich maps. Lateral connections then merge them with finer bottom-up maps, so every pyramid level carries strong semantics, and predictions are made separately on each level.

Plugged into a plain Faster R-CNN on the COCO benchmark, FPN set a new best single-model result, beating the entries of the COCO 2016 winners while running at about 6 FPS on a GPU. It also transfers to segmentation proposals. The work shows that explicit multi-scale handling still matters for deep networks and can be added at very little extra cost.

Key points

  • Compared with a strong single-scale Faster R-CNN baseline on ResNets, FPN improves:
    • proposal Average Recall by 8.0 points,
    • COCO-style AP by 2.3 points,
    • PASCAL-style AP by 3.8 points.
  • The pyramid is built from the last residual block of each ResNet stage (C2–C5, strides 4, 8, 16 and 32 pixels). conv1 is left out because of its memory cost.
  • Each merge step works as follows:
    • the coarser map is upsampled 2× with nearest-neighbor interpolation,
    • the bottom-up map passes through a 1×1 convolution,
    • the two are combined by element-wise addition.
  • For segmentation proposals in the DeepMask/SharpMask framework, FPN uses d = 128 and a 5×5 MLP predicting 14×14 masks, plus a 7×7 MLP for half-octave scales. It beats image-pyramid-based mask proposal methods by over 8.3 AR, nearly doubles accuracy on small objects, and runs at 6–7 FPS.
  • The mask model is trained end-to-end on 8 GPUs with a learning rate of 0.03 for 80k mini-batches, using 800-pixel images and no scale jitter. It uses no non-maximum suppression.

Interesting ideas

  • I1. Reuse the network's internal hierarchy instead of rescaling the input. The layers of a deep encoder already form a multi-resolution structure. Enriching that structure is far cheaper than running the model many times on resized inputs, and the same logic applies to any hierarchical encoder.
  • I2. Inject high-level information back into low-level, precise features. Adding upsampled deep features to shallow ones keeps the exact localization of the shallow maps and gains semantic strength. This pattern suits any task that needs both fine detail and global context.
  • I3. Predict at every level with one shared head. Making independent predictions on each pyramid level imitates a classic image pyramid, unlike designs that predict only from the finest merged map. Assigning objects to levels by size is a simple way to divide work across scales.
  • I4. Keep training and inference consistent. Because the pyramid is cheap, it can be used both in training and at test time. This avoids the mismatch created when multi-scale tricks are applied only at test time, a pitfall relevant to many pipelines.
  • I5. Cover in-between scales with a slightly larger head. A second head with a bigger receptive field (7×7 against 5×5, roughly √2 larger) handles half-octave scales without adding pyramid levels. This is a lightweight way to cover intermediate sizes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations🎓Cornell UniversityIthaca · United States134 works · 65,159 citations🎓Cornell TechNew York · United States9 works · 2,342 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan and Serge Belongie. 2016. Feature Pyramid Networks for Object Detection. arXiv:1612.03144.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1612.03144 · arXiv 1612.03144

@misc{lin2016feature,
  title = {{Feature Pyramid Networks for Object Detection}},
  author = {Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan and Serge Belongie},
  year = {2016},
  doi = {10.48550/arxiv.1612.03144},
  eprint = {1612.03144},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.03144}
}

References

40 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden. Pyramid methods in image processing. RCA engineer, 1984. find
  2. S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In CVPR, 2016. find
  3. Z. Cai, Q. Fan, R. S. Feris, and N. Vasconcelos. A unified multi-scale deep convolutional neural network for fast object detection. In ECCV, 2016. find
  4. J. Dai, K. He, Y. Li, S. Ren, and J. Sun. Instance-sensitive fully convolutional networks. In ECCV, 2016. find
  5. N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005. find
  6. P. Dollár, R. Appel, S. Belongie, and P. Perona. Fast feature pyramids for object detection. TPAMI, 2014. find
  7. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 2010. find
  8. G. Ghiasi and C. C. Fowlkes. Laplacian pyramid reconstruction and refinement for semantic segmentation. In ECCV, 2016. find
  9. S. Gidaris and N. Komodakis. Object detection via a multi-region & semantic segmentation-aware CNN model. In ICCV, 2015. find
  10. S. Gidaris and N. Komodakis. Attend refine repeat: Active box proposal generation via in-out localization. In BMVC, 2016. find
  11. R. Girshick. Fast R-CNN. In ICCV, 2015. find
  12. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
  13. B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and fine-grained localization. In CVPR, 2015. find
  14. K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. arXiv:1703.06870, 2017. find
  15. K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV. 2014. find
  16. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
  17. S. Honari, J. Yosinski, P. Vincent, and C. Pal. Recombinator networks: Learning coarse-to-fine feature aggregation. In CVPR, 2016. find
  18. T. Kong, A. Yao, Y. Chen, and F. Sun. Hypernet: Towards accurate region proposal generation and joint object detection. In CVPR, 2016. find
  19. A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
  20. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989. find
  21. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
  22. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed. SSD: Single shot multibox detector. In ECCV, 2016. find
  23. W. Liu, A. Rabinovich, and A. C. Berg. ParseNet: Looking wider to see better. In ICLR workshop, 2016. find
  24. J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. find
  25. D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004. find
  26. A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016. find
  27. P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS, 2015. find
  28. P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollár. Learning to refine object segments. In ECCV, 2016. find
  29. S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
  30. S. Ren, K. He, R. Girshick, X. Zhang, and J. Sun. Object detection networks on convolutional feature maps. PAMI, 2016. find
  31. O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In MIC-CAI, 2015.
  32. H. Rowley, S. Baluja, and T. Kanade. Human face detection in visual scenes. Technical Report CMU-CS-95-158R, Carnegie Mellon University, 1995. find
  33. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
  34. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In ICLR, 2014.
  35. A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In CVPR, 2016. find
  36. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  37. J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. IJCV, 2013. find
  38. R. Vaillant, C. Monrocq, and Y. LeCun. Original approach for the localisation of objects in images. IEE Proc. on Vision, Image, and Signal Processing, 1994. find
  39. S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, 2016. find
  40. S. Zagoruyko, A. Lerer, T.-Y. Lin, P. O. Pinheiro, S. Gross, S. Chintala, and P. Dollár. A multipath network for object detection. In BMVC, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2016-feature-pyramid-networks-object-detection/