Elamur Research
☰
home / papers / 2017 / What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1703.04977

Summary

Kendall and Gal ask which kind of uncertainty matters most for deep vision models. They distinguish aleatoric uncertainty, which is noise in the observations themselves, from epistemic uncertainty, which comes from not knowing the model parameters and shrinks as more data arrives.

The authors propose a single Bayesian deep learning framework that combines both:

  • Aleatoric part: the network predicts an input-dependent (heteroscedastic) noise term alongside its output.
  • Epistemic part: Monte Carlo dropout approximates the posterior over the weights.

They derive versions for regression (per-pixel depth) and classification (semantic segmentation), testing on CamVid, NYUv2 and Make3D. Modelling the noise explicitly produces new loss functions that act as learned attenuation, so noisy pixels count less during training. The combined models set new state-of-the-art results on segmentation and depth benchmarks.

The analysis shows a practical trade-off. Aleatoric uncertainty is cheap to compute and most useful when data is abundant. Epistemic uncertainty is essential for spotting inputs unlike the training data, which matters in safety-critical settings, but its sampling cost makes real-time use difficult.

Key points

  • Learned loss attenuation from aleatoric modelling improves results by about 1–3% over non-Bayesian baselines.
  • Shrinking the training set changes epistemic and aleatoric uncertainty very differently:
    • On CamVid, using 1/4, 1/2 and the full training set gives segmentation IoU of 57.2, 62.9 and 67.5.
    • On Make3D, the same subsets give depth RMS of 5.76, 4.62 and 3.87.
    • Epistemic uncertainty falls as training data grows, while aleatoric uncertainty stays roughly flat.
  • On out-of-distribution test data (NYUv2 tested with CamVid- or Make3D-trained models), epistemic uncertainty rises sharply. Aleatoric uncertainty alone cannot flag such inputs.
  • Aleatoric uncertainty concentrates on object boundaries and distant objects. Epistemic uncertainty concentrates on semantically or visually hard pixels.
  • Speed:
    • A DenseNet-based model processes a 640×480 image in about 150 ms on an NVIDIA Titan X, and aleatoric heads add negligible cost.
    • 50 Monte Carlo dropout samples cause roughly a 50× slowdown for DenseNet.
    • The authors name real-time epistemic uncertainty as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.04977 · arXiv 1703.04977

@misc{kendall2017what,
  title = {{What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?}},
  author = {Kendall, Alex and Gal, Yarin},
  year = {2017},
  doi = {10.48550/arxiv.1703.04977},
  eprint = {1703.04977},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.04977}
}

References

36 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. NHTSA. PE 16-007. Technical report, U.S. Department of Transportation, National Highway Traffic Safety Administration, Jan 2017. Tesla Crash Preliminary Evaluation Report. find
  2. Jessica Guynn. Google photos labeled black people ’gorillas’. USA Today, 2015. find
  3. Andrew Blake, Rupert Curwen, and Andrew Zisserman. A framework for spatiotemporal control in the tracking of visual contours. International Journal of Computer Vision, 11(2):127–145, 1993. find
  4. Xuming He, Richard S Zemel, and Miguel Á Carreira-Perpiñán. Multiscale conditional random fields for image labeling. In Computer vision and pattern recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE computer society conference on, volume 2, pages II–II. IEEE, 2004. find
  5. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. find
  6. Y. Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016. find
  7. Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural Safety, 31 (2):105–112, 2009. find
  8. Gabriel J Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, 30(2):88–97, 2009. find
  9. John Denker and Yann LeCun. Transforming neural-net output levels to probability distributions. In Advances in Neural Information Processing Systems 3. Citeseer, 1991. find
  10. David JC MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472, 1992. find
  11. Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995. find
  12. Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pages 2348–2356, 2011. find
  13. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ICML, 2015. find
  14. José Miguel Hernández-Lobato, Yingzhen Li, Daniel Hernández-Lobato, Thang Bui, and Richard E Turner. Black-box alpha divergence minimization. In Proceedings of The 33rd International Conference on Machine Learning, pages 1511–1520, 2016. find
  15. Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with Bernoulli approximate variational inference. ICLR workshop track, 2016. find
  16. Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999. find
  17. David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability distribution. In Neural Networks, 1994. IEEE World Congress on Computational Intelligence., 1994 IEEE International Conference On, volume 1, pages 55–60. IEEE, 1994. find
  18. Quoc V Le, Alex J Smola, and Stéphane Canu. Heteroscedastic Gaussian process regression. In Proceedings of the 22nd international conference on Machine learning, pages 489–496. ACM, 2005. find
  19. Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016. — [arXiv:1608.06993](https://arxiv.org/abs/1608.06993) find
  20. Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. arXiv preprint arXiv:1611.09326, 2016. — [arXiv:1611.09326](https://arxiv.org/abs/1611.09326) find
  21. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation… find
  22. Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. Bayesian SegNet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv preprint arXiv:1511.02680, 2015. — [arXiv:1511.02680](https://arxiv.org/abs/1511.02680) find
  23. Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, pages 746–760. Springer, 2012. find
  24. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062, 2014. — [arXiv:1412.7062](https://arxiv.org/abs/1412.7062) find
  25. Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence, 31(5):824–840, 2009. find
  26. Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 3D Vision (3DV), 2016 Fourth International Conference on, pages 239–248. IEEE, 2016. find
  27. David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems, pages 2366–2374, 2014. find
  28. Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A deep convolutional encoder-decoder architecture for scene segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. find
  29. Evan Shelhamer, Jonathon Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, 2016. find
  30. Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015. — [arXiv:1511.07122](https://arxiv.org/abs/1511.07122) find
  31. Abhijit Kundu, Vibhav Vineet, and Vladlen Koltun. Feature space optimization for semantic video segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3168–3175, 2016. find
  32. David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE International Conference on Computer Vision, pages 2650–2658, 2015. find
  33. Kevin Karsch, Ce Liu, and Sing Bing Kang. Depth extraction from video using non-parametric sampling. In European Conference on Computer Vision, pages 775–788. Springer, 2012. find
  34. Miaomiao Liu, Mathieu Salzmann, and Xuming He. Discrete-continuous depth estimation from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 716–723, 2014. find
  35. Bo Li, Chunhua Shen, Yuchao Dai, Anton van den Hengel, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1119–1127, 2015. find
  36. Lubor Ladicky, Jianbo Shi, and Marc Pollefeys. Pulling things out of perspective. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 89–96, 2014. find