Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1506.01497

Summary

By 2015, detectors such as SPPnet and Fast R-CNN had become fast enough that generating candidate object regions was the main cost at test time. These candidates usually came from hand-engineered CPU methods like Selective Search or EdgeBoxes. The authors replace that external step with a Region Proposal Network (RPN). The RPN is a small fully convolutional head that reuses the convolutional feature map the detector already computes. At every location, it scores and refines a set of "anchor" boxes with several scales and aspect ratios. Multi-scale handling therefore comes from reference boxes rather than image or filter pyramids. An alternating fine-tuning scheme lets the RPN and Fast R-CNN share convolutional layers, which yields one unified detector, Faster R-CNN. In it, the RPN works like an attention module that points to promising regions. On PASCAL VOC 2007/2012 and MS COCO, the system beats the Selective Search + Fast R-CNN baseline while making proposals nearly free. This made accurate, near-real-time deep detection practical and became a foundation for many later systems.

Key points

• Speed: Proposal cost drops to about 10 ms per image, compared with about 2 s for Selective Search (CPU) and 0.2 s for EdgeBoxes. The full VGG-16 pipeline runs at 5 fps on a GPU with only 300 proposals per image.
• RPN design: A 3×3 window slides over the last shared conv layer and produces a 256-d (ZF) or 512-d (VGG-16) feature. Two sibling 1×1 layers follow: reg, which outputs 4k box coordinates, and cls, which outputs 2k object/not-object scores for k anchors.
• Backbones: The authors test ZF (5 shareable conv layers) and VGG-16 (13 shareable conv layers). The window's effective receptive field is 171 px and 228 px respectively.
• Impact: Faster R-CNN and RPN, combined with 101-layer ResNets, underpinned 1st-place entries at ILSVRC and COCO 2015 in ImageNet detection and localization and COCO detection and segmentation. The system was also deployed at Pinterest.
• Code and training: Code is released in MATLAB (shaoqingren/faster_rcnn) and Python (rbgirshick/py-faster-rcnn). The authors also found that joint RPN + Fast R-CNN training works and shortens training time.