End-to-End Object Detection with Transformers

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2005.12872

Summary

Conventional object detectors solve detection indirectly. They regress and classify relative to proposals, anchors or grid centers, then depend on hand-tuned parts such as anchor design and non-maximum suppression (NMS) to remove duplicates. Carion et al. (Facebook AI) instead frame detection as predicting a set directly. Their model, DETR (DEtection TRansformer), combines a standard CNN backbone (ResNet) with a transformer encoder-decoder. Starting from a small fixed set of learned object queries, it emits all detections in parallel. Training relies on a set-level loss: the Hungarian algorithm computes a one-to-one assignment between predictions and ground-truth boxes, and leftover slots must output a "no object" class. On COCO, DETR matches a strong, heavily tuned Faster R-CNN baseline in accuracy and speed. It is notably better on large objects but weaker on small ones, and it needs a very long training schedule. A lightweight mask head extends it to panoptic segmentation, where it outperforms established baselines. The work matters because it removes much detection-specific engineering and requires no custom layers.

Key points

Main components: a bipartite-matching (Hungarian) loss over a fixed set of N predictions, plus non-autoregressive parallel decoding with a transformer. The design uses no anchors and no NMS.
In the loss, the classification term for "no object" slots is down-weighted by a factor of 10 to offset class imbalance. Auxiliary decoding losses help training.
For panoptic segmentation on COCO (80 things + 53 stuff categories), a mask head is trained for 25 epochs on top of a frozen DETR. Per-pixel argmax merges the masks without overlap heuristics.
On COCO val, DETR-R101 reaches 45.1 PQ versus 44.1 for a retrained PanopticFPN++ (R101), and 37.0 versus 33.6 PQ on stuff classes. It reaches 46 PQ on the COCO test set, although its mask AP trails the baselines by up to 8 points.
Training code and pretrained models are released at github.com/facebookresearch/detr.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

344 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 333 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.12872 · arXiv 2005.12872

@misc{carion2020end,
  title = {{End-to-End Object Detection with Transformers}},
  author = {Carion, Nicolas and Massa, Francisco and Synnaeve, Gabriel and Usunier, Nicolas and Kirillov, Alexander and Zagoruyko, Sergey},
  year = {2020},
  doi = {10.48550/arxiv.2005.12872},
  eprint = {2005.12872},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.12872}
}

References

3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Al-Rfou, R., Choe, D., Constant, N., Guo, M., Jones, L.: Character-level language modeling with deeper self-attention. In: AAAI Conference on Artificial Intelligence (2019) 2. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)… find
PAMI (2019) 6. Chan, W., Saharia, C., Hinton, G., Norouzi, M., Jaitly, N.: Imputer: Sequence modelling via imputation and dynamic programming. arXiv:2002.08926 (2020) 7. Cordonnier, J.B., Loukas, A., Jaggi, M.: On the relationship between self-attention and convolutional layers. In: ICLR (2020) 8. D… find
PAMI (2015) 38. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union. In: CVPR (2019) 39. Rezatofighi, S.H., Kaskman, R., Motlagh, F.T., Shi, Q., Cremers, D., Leal-Taixé, L., Reid, I.: Deep perm-set net: Learn to predict sets with unknown p… find