Elamur Research⌕
☰
home / papers / 2021 / A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

Paper · 2021 · Anastasios N. Angelopoulos and Stephen Bates

Summary

Modern machine learning models that operate as a "black box" are increasingly deployed in high-stakes domains such as medicine. This calls for a reliable assessment of how confident a model is. The authors write a tutorial-style overview of conformal prediction. This approach turns any heuristic uncertainty measure of an already trained model into a prediction set or interval with a rigorous statistical guarantee. All it requires is a small held-out calibration set: a conformal score is computed on it, and its adjusted quantile is taken. The guarantee does not rely on assumptions about the data distribution or the quality of the model. It only requires the calibration and test examples to be exchangeable (in particular, i.i.d.). The tutorial combines theory with examples for classification, regression, segmentation, time series, outlier detection, and distribution shift. Each example comes with Python code and a Jupyter notebook on real data. The work matters because it makes rigorous uncertainty quantification accessible to practitioners in NLP, computer vision, and other fields.

Key points

  • Coverage guarantee: the prediction set contains the correct answer with probability at least 1 − α. This probability is bounded above by 1 − α + 1/(n+1). For example, with α = 0.1 the coverage is about 90%. The main variant of the method the authors consider is split conformal prediction.
  • Simplicity: calibration takes only a few lines of code and about 500 calibration examples. In the basic classification example on Imagenet, the score equals one minus the softmax probability of the correct class.
  • Extensions: Adaptive Prediction Sets, conformalized quantile regression, conformal risk control, class- and group-conditional coverage, covariate shift, distribution drift, full conformal, CV+ and Jackknife+, as well as the Learn then Test approach for controlling general risks.
  • Worked examples: multilabel classification, tumor segmentation, weather forecasting under temporal shift, detecting toxic comments as outliers, selective classification.
  • Applications and resources: The Washington Post used the method to estimate not-yet-counted votes in the 2020 US elections. The authors also mention the scikit-learn-compatible MAPIE library and the Awesome Conformal Prediction repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Anastasios N. Angelopoulos and Stephen Bates. 2021. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2107.07511 · arXiv 2107.07511

@misc{angelopoulos2021gentle,
  title = {{A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification}},
  author = {Anastasios N. Angelopoulos and Stephen Bates},
  year = {2021},
  doi = {10.48550/arxiv.2107.07511},
  eprint = {2107.07511},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2107.07511}
}

References

132 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. V. Vovk, A. Gammerman, and G. Shafer, Algorithmic Learning in a Random World. Springer, 2005. find
  2. H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman, “Inductive confidence machines for regression,” in Machine Learning: European Conference on Machine Learning, 2002, pp. 345–356. find
  3. J. Lei and L. Wasserman, “Distribution-free prediction bands for non-parametric regression,” Journal of the Royal Statistical Society: Series B: Statistical Methodology, pp. 71–96, 2014. find
  4. A. N. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan, “Uncertainty sets for image classifiers using conformal prediction,” in International Conference on Learning Representations, 2021. find
  5. V. Vovk, A. Gammerman, and C. Saunders, “Machine-learning applications of algorithmic randomness,” in International Conference on Machine Learning, 1999, pp. 444–453. find
  6. M. Sadinle, J. Lei, and L. Wasserman, “Least ambiguous set-valued classifiers with bounded error levels,” Journal of the American Statistical Association, vol. 114, pp. 223–234, 2019. find
  7. Y. Romano, M. Sesia, and E. J. Candès, “Classification with valid and adaptive coverage,” arXiv:2006.02544, 2020. — [arXiv:2006.02544](https://arxiv.org/abs/2006.02544) find
  8. Y. Romano, E. Patterson, and E. Candès, “Conformalized quantile regression,” in Advances in Neural Information Processing Systems, vol. 32, 2019, pp. 3543–3553. find
  9. R. Koenker and G. Bassett Jr, “Regression quantiles,” Econometrica: Journal of the Econometric Society, vol. 46, no. 1, pp. 33–50, 1978. find
  10. A. N. Angelopoulos, A. P. Kohli, S. Bates, M. I. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula, and Y. Romano, “Image-to-image regression with distribution-free uncertainty quantification and applications in imaging,” arXiv preprint arXiv:2202.05265, 2022. — [arXiv:2202.05265](https://arxiv.org/abs/2202.05265) find
  11. P. Hoff, “Bayes-optimal prediction with frequentist coverage control,” arXiv:2105.14045, 2021. — [arXiv:2105.14045](https://arxiv.org/abs/2105.14045) find
  12. L. Wasserman, “Frasian inference,” Statistical Science, vol. 26, no. 3, pp. 322–325, 2011. find
  13. T. Melluish, C. Saunders, I. Nouretdinov, and V. Vovk, “Comparing the bayes and typicalness frameworks,” in European Conference on Machine Learning, Springer, 2001, pp. 360–371. find
  14. V. Vovk, “Conditional validity of inductive conformal predictors,” in Proceedings of the Asian Conference on Machine Learning, vol. 25, 2012, pp. 475–490. find
  15. M. Cauchois, S. Gupta, and J. Duchi, “Knowing what you know: Valid and validated confidence sets in multiclass and multilabel prediction,” arXiv:2004.10181, 2020. — [arXiv:2004.10181](https://arxiv.org/abs/2004.10181) find
  16. S. Feldman, S. Bates, and Y. Romano, “Improving conditional coverage via orthogonal quantile regression,” in Advances in Neural Information Processing Systems, 2021. find
  17. A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,” arXiv preprint arXiv:2208.02814, 2022. — [arXiv:2208.02814](https://arxiv.org/abs/2208.02814) find
  18. A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei, “Learn then test: Calibrating predictive algorithms to achieve risk control,” arXiv:2110.01052, 2021. — [arXiv:2110.01052](https://arxiv.org/abs/2110.01052) find
  19. M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko, “A review of novelty detection,” Signal Processing, vol. 99, pp. 215–249, 2014. find
  20. R. A. Fisher, “Design of experiments,” British Medical Journal, vol. 1, no. 3923, p. 554, 1936. find
  21. E. J. Pitman, “Significance tests which may be applied to samples from any populations,” Supplement to the Journal of the Royal Statistical Society, vol. 4, no. 1, pp. 119–130, 1937. find
  22. V. Vovk, I. Nouretdinov, and A. Gammerman, “Testing exchangeability on-line,” in Proceedings of the 20th International Conference on Machine Learning (ICML-03), 2003, pp. 768–775. find
  23. L. Guan and R. Tibshirani, “Prediction and outlier detection in classification problems,” arXiv:1905.04396, 2019. — [arXiv:1905.04396](https://arxiv.org/abs/1905.04396) find
  24. S. Bates, E. Candès, L. Lei, Y. Romano, and M. Sesia, “Testing for outliers with conformal p-values,” arXiv:2104.08279, 2021. — [arXiv:2104.08279](https://arxiv.org/abs/2104.08279) find
  25. R. J. Tibshirani, R. Foygel Barber, E. Candes, and A. Ramdas, “Conformal prediction under covariate shift,” in Advances in Neural Information Processing Systems 32, 2019, pp. 2530–2540. find
  26. R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond exchangeability,” arXiv:2202.13415, 2022. — [arXiv:2202.13415](https://arxiv.org/abs/2202.13415) find
  27. L. Guan, “Conformal prediction with localization,” arXiv:1908.08558, 2020. — [arXiv:1908.08558](https://arxiv.org/abs/1908.08558) find
  28. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision, Springer, 2014, pp. 740–755.
  29. A. Malinin, N. Band, G. Chesnokov, Y. Gal, M. J. Gales, A. Noskov, A. Ploskonosov, L. Prokhorenkova, I. Provilkov, V. Raina, et al., “Shifts: A dataset of real distributional shift across multiple large-scale tasks,” arXiv preprint arXiv:2107.07455, 2021. — [arXiv:2107.07455](https://arxiv.org/abs/2107.07455) find
  30. A. V. Dorogush, V. Ershov, and A. Gulin, “Catboost: Gradient boosting with categorical features support,” arXiv preprint arXiv:1810.11363, 2018. find
  31. I. Gibbs and E. Candès, “Adaptive conformal inference under distribution shift,” arXiv:2106.00170, 2021. — [arXiv:2106.00170](https://arxiv.org/abs/2106.00170) find
  32. M. Zaffran, O. Féron, Y. Goude, J. Josse, and A. Dieuleveut, “Adaptive conformal predictions for time series,” in International Conference on Machine Learning, PMLR, 2022, pp. 25 834–25 866. find
  33. I. Gibbs and E. Candès, “Conformal inference for online prediction with arbitrary distribution shifts,” arXiv preprint arXiv:2208.08401, 2022. — [arXiv:2208.08401](https://arxiv.org/abs/2208.08401) find
  34. C. Xu and Y. Xie, “Conformal prediction interval for dynamic time-series,” in International Conference on Machine Learning, PMLR, 2021, pp. 11 559–11 569. find
  35. L. Hanu and Unitary team, Detoxify, Github. https://github.com/unitaryai/detoxify, 2020. ↗
  36. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  37. P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, et al., “Wilds: A benchmark of in-the-wild distribution shifts,” in International Conference on Machine Learning, PMLR, 2021, pp. 5637–5664. find
  38. G. Shafer and V. Vovk, “A tutorial on conformal prediction,” Journal of Machine Learning Research, vol. 9, no. Mar, pp. 371–421, 2008. find
  39. E. Ndiaye and I. Takeuchi, “Computing full conformal prediction set with approximate homotopy,” in Advances in Neural Information Processing Systems, 2019. find
  40. E. Ndiaye and I. Takeuchi, “Root-finding approaches for computing conformal prediction set,” Machine Learning, 2022. find
  41. V. Vovk, “Cross-conformal predictors,” Annals of Mathematics and Artificial Intelligence, vol. 74, no. 1-2, pp. 9–28, 2015. find
  42. R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani, “Predictive inference with the jackknife+,” The Annals of Statistics, vol. 49, no. 1, pp. 486–507, 2021. find
  43. E. Chung and J. P. Romano, “Exact and asymptotically robust permutation tests,” The Annals of Statistics, vol. 41, no. 2, pp. 484–507, 2013. find
  44. H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” The Annals of Mathematical Statistics, pp. 50–60, 1947. find
  45. E. L. Lehmann, “The power of rank tests,” The Annals of Mathematical Statistics, pp. 23–43, 1953. find
  46. Z. Sidak, P. K. Sen, and J. Hajek, Theory of rank tests. Elsevier, 1999. find
  47. B. Efron and R. J. Tibshirani, An introduction to the bootstrap. CRC press, 1994. find
  48. S. Chatterjee and P. Qiu, “Distribution-free cumulative sum control charts using bootstrap-based control limits,” The Annals of Applied Statistics, vol. 3, no. 1, pp. 349–369, 2009. find
  49. G. T. Fechner, Kollektivmasslehre. Engelmann, 1897. find
  50. R. von Mises, “Grundlagen der wahrscheinlichkeitsrechnung,” Mathematische Zeitschrift, vol. 5, no. 1, pp. 52–99, 1919. find
  51. A. Wald, “Die widerspruchfreiheit des kollectivbegriffes der wahrscheinlichkeitsrechnung,” Ergebnisse Eines Mathematischen Kolloquiums, vol. 8, no. 38-72, p. 37, 1937. find
  52. A. Church, “On the concept of a random sequence,” Bulletin of the American Mathematical Society, vol. 46, no. 2, pp. 130–135, 1940. find
  53. J. Ville, “Etude critique de la notion de collectif,” Bull. Amer. Math. Soc, vol. 45, no. 11, p. 824, 1939. find
  54. G. Shafer and V. Vovk, “The sources of Kolmogorov’s Grundbegriffe,” Statistical Science, vol. 21, no. 1, pp. 70–98, 2006. find
  55. V. Vovk, “Kolmogorov’s complexity conception of probability,” Synthese Library, pp. 51–70, 2001. find
  56. C. P. Porter, “Kolmogorov on the role of randomness in probability theory,” Mathematical Structures in Computer Science, vol. 24, no. 3, 2014. find
  57. A. N. Kolmogorov, “Three approaches to the quantitative definition of information,” Problems of Information Transmission, vol. 1, no. 1, pp. 1–7, 1965. find
  58. A. Kolmogorov, “Logical basis for information theory and probability theory,” IEEE Transactions on Information Theory, vol. 14, no. 5, pp. 662–664, 1968. find
  59. A. N. Kolmogorov, “Combinatorial foundations of information theory and the calculus of probabilities,” Russian Mathematical Surveys, vol. 38, no. 4, pp. 29–40, 1983. find
  60. V. G. Vovk, “On the concept of the Bernoulli property,” Russian Mathematical Surveys, vol. 41, no. 1, p. 247, 1986. find
  61. V. Vovk, “Testing randomness online,” Statistical Science, vol. 36, no. 4, pp. 595–611, 2021. find
  62. F. Mota, S. Aaronson, L. Antunes, and A. Souto, “Sophistication as randomness deficiency,” in International Workshop on Descriptional Complexity of Formal Systems, Springer, 2013, pp. 172– 181. find
  63. S. S. Wilks, “Determination of sample sizes for setting tolerance limits,” Annals of Mathematical Statistics, vol. 12, no. 1, pp. 91–96, 1941. find
  64. ——, “Statistical prediction with special reference to the problem of tolerance limits,” Annals of Mathematical Statistics, vol. 13, no. 4, pp. 400–409, 1942. find
  65. A. Wald, “An extension of Wilks’ method for setting tolerance limits,” Annals of Mathematical Statistics, vol. 14, no. 1, pp. 45–55, 1943. find
  66. J. W. Tukey, “Non-parametric estimation II. Statistically equivalent blocks and tolerance regions–the continuous case,” Annals of Mathematical Statistics, vol. 18, no. 4, pp. 529–539, 1947. find
  67. P. Diaconis and D. Freedman, “Finite exchangeable sequences,” The Annals of Probability, pp. 745– 764, 1980. find
  68. D. J. Aldous, “Exchangeability and related topics,” in École d’Été de Probabilités de Saint-Flour XIII—1983, 1985, pp. 1–198. find
  69. B. De Finetti, “Funzione caratteristica di un fenomeno aleatorio,” in Atti del Congresso Internazionale dei Matematici: Bologna del 3 al 10 de Settembre di 1928, 1929, pp. 179–190. find
  70. D. A. Freedman, “Bernard Friedman’s urn,” The Annals of Mathematical Statistics, pp. 956–970, 1965. find
  71. E. Hewitt and L. J. Savage, “Symmetric measures on Cartesian products,” Transactions of the American Mathematical Society, vol. 80, no. 2, pp. 470–501, 1955. find
  72. J. F. Kingman, “Uses of exchangeability,” The Annals of Probability, vol. 6, no. 2, pp. 183–197, 1978. find
  73. E. Dobriban, Topics in Modern Statistical Learning (STAT 991, UPenn, 2022 Spring), Dec. 2022. find
  74. A. Gammerman, V. Vovk, and V. Vapnik, “Learning by transduction,” Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, vol. 14, pp. 148–155, 1998. find
  75. C. Saunders, A. Gammerman, and V. Vovk, “Transduction with confidence and credibility,” 1999. find
  76. V. Vovk, “On-line confidence machines are well-calibrated,” in The 43rd Annual IEEE Symposium on Foundations of Computer Science, IEEE, 2002, pp. 187–196. find
  77. V. Vovk, G. Shafer, and I. Nouretdinov, “Self-calibrating probability forecasting.,” in Neural Information Processing Systems, 2003, pp. 1133–1140. find
  78. V. Vovk and I. Petej, “Venn-Abers predictors,” arXiv:1211.0025, 2012. — [arXiv:1211.0025](https://arxiv.org/abs/1211.0025) find
  79. V. Vovk, J. Shen, V. Manokhin, and M.-g. Xie, “Nonparametric predictive distributions based on conformal prediction,” Machine Learning, pp. 1–30, 2017. find
  80. J. Lei, J. Robins, and L. Wasserman, “Efficient nonparametric conformal prediction regions,” arXiv:1111.1418, 2011. — [arXiv:1111.1418](https://arxiv.org/abs/1111.1418) find
  81. ——, “Distribution-free prediction sets,” Journal of the American Statistical Association, vol. 108, no. 501, pp. 278–287, 2013. find
  82. B. Póczos, A. Singh, A. Rinaldo, and L. Wasserman, “Distribution-free distribution regression,” in Artificial Intelligence and Statistics, PMLR, 2013, pp. 507–515. find
  83. J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman, “Distribution-free predictive inference for regression,” Journal of the American Statistical Association, vol. 113, no. 523, pp. 1094– 1111, 2018. find
  84. J. Lei, A. Rinaldo, and L. Wasserman, “A conformal prediction approach to explore functional data,” Annals of Mathematics and Artificial Intelligence, vol. 74, pp. 29–43, 2015. find
  85. J. Lei, “Fast exact conformalization of the lasso using piecewise linear homotopy,” Biometrika, vol. 106, no. 4, pp. 749–764, 2019. find
  86. C. Gupta, A. K. Kuchibhotla, and A. Ramdas, “Nested conformal prediction and quantile out-of-bag ensemble methods,” Pattern Recognition, p. 108 496, 2021. find
  87. R. Foygel Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani, “The limits of distribution-free conditional predictive inference,” Information and Inference: A Journal of the IMA, vol. 10, no. 2, pp. 455–482, 2021. find
  88. Y. Lee and R. F. Barber, “Distribution-free inference for regression: Discrete, continuous, and in between,” arXiv:2105.14075, 2021. — [arXiv:2105.14075](https://arxiv.org/abs/2105.14075) find
  89. R. Izbicki, G. Shimizu, and R. Stern, “Flexible distribution-free conditional predictive bands using density estimators,” in Proceedings of Machine Learning Research, vol. 108, PMLR, 2020, pp. 3068– 3077. find
  90. J. Lei, “Classification with confidence,” Biometrika, vol. 101, no. 4, pp. 755–769, Oct. 2014. find
  91. Y. Hechtlinger, B. Poczos, and L. Wasserman, “Cautious deep learning,” arXiv:1805.09460, 2018. — [arXiv:1805.09460](https://arxiv.org/abs/1805.09460) find
  92. D. Stutz, K. D. Dvijotham, A. T. Cemgil, and A. Doucet, “Learning optimal conformal classifiers,” in International Conference on Learning Representations, 2022. find
  93. M. Cauchois, S. Gupta, A. Ali, and J. C. Duchi, “Robust validation: Confident predictions even when distributions shift,” arXiv:2008.04267, 2020. — [arXiv:2008.04267](https://arxiv.org/abs/2008.04267) find
  94. L. Lei and E. J. Candès, “Conformal inference of counterfactuals and individual treatment effects,” arXiv:2006.06138, 2020. — [arXiv:2006.06138](https://arxiv.org/abs/2006.06138) find
  95. M. Yin, C. Shi, Y. Wang, and D. M. Blei, “Conformal sensitivity analysis for individual treatment effects,” arXiv:2112.03493, 2021. find
  96. V. Chernozhukov, K. Wüthrich, and Y. Zhu, “An exact and robust conformal inference method for counterfactual and synthetic controls,” Journal of the American Statistical Association, pp. 1–16, 2021. find
  97. E. J. Candès, L. Lei, and Z. Ren, “Conformalized survival analysis,” arXiv:2103.09763, 2021. — [arXiv:2103.09763](https://arxiv.org/abs/2103.09763) find
  98. A. N. Angelopoulos, S. Bates, T. Zrnic, and M. I. Jordan, “Private prediction sets,” arXiv:2102.06202, 2021. — [arXiv:2102.06202](https://arxiv.org/abs/2102.06202) find
  99. V. Chernozhukov, K. Wüthrich, and Z. Yinchu, “Exact and robust conformal inference methods for predictive machine learning with dependent data,” in Conference On Learning Theory, PMLR, 2018, pp. 732–749. find
  100. R. Dunn, L. Wasserman, and A. Ramdas, “Distribution-free prediction sets with random effects,” arXiv:1809.07441, 2018. — [arXiv:1809.07441](https://arxiv.org/abs/1809.07441) find
  101. R. I. Oliveira, P. Orenstein, T. Ramos, and J. V. Romano, “Split conformal prediction for dependent data,” arXiv:2203.15885, 2022. — [arXiv:2203.15885](https://arxiv.org/abs/2203.15885) find
  102. O. Bastani, V. Gupta, C. Jung, G. Noarov, R. Ramalingam, and A. Roth, “Practical adversarial multivalid conformal prediction,” in Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. find
  103. C. Jung, G. Noarov, R. Ramalingam, and A. Roth, “Batch multivalid conformal prediction,” arXiv preprint arXiv:2209.15145, 2022. — [arXiv:2209.15145](https://arxiv.org/abs/2209.15145) find
  104. C. Gupta and A. Ramdas, “Distribution-free calibration guarantees for histogram binning without sample splitting,” in International Conference on Machine Learning, vol. 139, 2021, pp. 3942–3952. find
  105. S. Park, S. Li, O. Bastani, and I. Lee, “PAC confidence predictions for deep neural network classifiers,” in International Conference on Learning Representations, 2021. find
  106. D. Volkhonskiy, E. Burnaev, I. Nouretdinov, A. Gammerman, and V. Vovk, “Inductive conformal martingales for change-point detection,” in Conformal and Probabilistic Prediction and Applications, PMLR, 2017, pp. 132–153. find
  107. X. Hu and J. Lei, “A distribution-free test of covariate shift using conformal prediction,” arXiv:2010.07147, 2020. — [arXiv:2010.07147](https://arxiv.org/abs/2010.07147) find
  108. A. Podkopaev and A. Ramdas, “Tracking the risk of a deployed model and detecting harmful distribution shifts,” arXiv:2110.06177, 2021. — [arXiv:2110.06177](https://arxiv.org/abs/2110.06177) find
  109. P. Chaudhuri, “Global nonparametric estimation of conditional quantile functions and their derivatives,” Journal of Multivariate Analysis, vol. 39, no. 2, pp. 246–269, 1991. find
  110. I. Steinwart and A. Christmann, “Estimating conditional quantiles with the help of the pinball loss,” Bernoulli, vol. 17, no. 1, pp. 211–225, 2011. find
  111. I. Takeuchi, Q. V. Le, T. D. Sears, and A. J. Smola, “Nonparametric quantile estimation,” Journal of Machine Learning Research, vol. 7, pp. 1231–1264, 2006. find
  112. K. Q. Zhou, S. L. Portnoy, et al., “Direct use of regression quantiles to construct confidence sets in linear models,” The Annals of Statistics, vol. 24, no. 1, pp. 287–306, 1996. find
  113. K. Q. Zhou and S. L. Portnoy, “Statistical inference on heteroscedastic models based on regression quantiles,” Journal of Nonparametric Statistics, vol. 9, no. 3, pp. 239–260, 1998. find
  114. R. Koenker, Quantile Regression. Cambridge University Press, 2005. find
  115. R. Koenker, V. Chernozhukov, X. He, and L. Peng, “Handbook of quantile regression,” 2018. find
  116. R. Koenker, “Additive models for quantile regression: Model selection and confidence bandaids,” Brazilian Journal of Probability and Statistics, vol. 25, no. 3, pp. 239–262, 2011. find
  117. A. Fisch, T. Schuster, T. S. Jaakkola, and R. Barzilay, “Efficient conformal prediction via cascaded inference with expanded admission,” in International Conference on Learning Representations, 2021. find
  118. T. Schuster, A. Fisch, T. Jaakkola, and R. Barzilay, “Consistent accelerated inference via confident adaptive transformers,” Empirical Methods in Natural Language Processing, 2021. find
  119. A. Fisch, T. Schuster, T. Jaakkola, and D. Barzilay, “Few-shot conformal prediction with auxiliary tasks,” in International Conference on Machine Learning, vol. 139, 2021, pp. 3329–3339. find
  120. U. Johansson, H. Boström, T. Löfström, and H. Linusson, “Regression conformal prediction with random forests,” Machine learning, vol. 97, no. 1, pp. 155–176, 2014. find
  121. H. Linusson, U. Norinder, H. Boström, U. Johansson, and T. Löfström, “On the calibration of aggregated conformal predictors,” in Conformal and probabilistic prediction and applications, PMLR, 2017, pp. 154–173. find
  122. H. Boström, H. Linusson, T. Löfström, and U. Johansson, “Accelerating difficulty estimation for conformal regression forests,” Annals of Mathematics and Artificial Intelligence, vol. 81, no. 1, pp. 125– 144, 2017. find
  123. J. Cherian and L. Bronner, “How the Washington Post estimates outstanding votes for the 2020 presidential election,” Washington Post, 2021, https://s3.us-east-1.amazonaws.com/elex-models-prod/2020-general/write-up/election model writeup.pdf. ↗
  124. C. Lu and J. Kalpathy-Cramer, “Distribution-free federated learning with conformal predictions,” arXiv:2110.07661, 2021. — [arXiv:2110.07661](https://arxiv.org/abs/2110.07661) find
  125. C. Lu, A. Lemay, K. Chang, K. Hoebel, and J. Kalpathy-Cramer, “Fair conformal predictors for applications in medical imaging,” arXiv:2109.04392, 2021. — [arXiv:2109.04392](https://arxiv.org/abs/2109.04392) find
  126. Y. Romano, R. F. Barber, C. Sabatti, and E. Candès, “With malice toward none: Assessing uncertainty via equalized coverage,” Harvard Data Science Review, vol. 2, no. 2, Apr. 30, 2020. find
  127. A. K. Kuchibhotla and R. A. Berk, “Nested conformal prediction sets for classification with applications to probation data,” arXiv:2104.09358, 2021. — [arXiv:2104.09358](https://arxiv.org/abs/2104.09358) find
  128. L. Lindemann, M. Cleaveland, G. Shim, and G. J. Pappas, “Safe planning in dynamic environments using conformal prediction,” arXiv preprint arXiv:2210.10254, 2022. — [arXiv:2210.10254](https://arxiv.org/abs/2210.10254) find
  129. A. Dixit, L. Lindemann, M. Cleaveland, S. Wei, G. J. Pappas, and J. W. Burdick, “Adaptive conformal prediction for motion planning among dynamic agents,” arXiv preprint arXiv:2212.00278, 2022. — [arXiv:2212.00278](https://arxiv.org/abs/2212.00278) find
  130. V. Manokhin, Awesome Conformal Prediction, version v1.0.0, Apr. 2022. find
  131. S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. Jordan, “Distribution-free, risk-controlling prediction sets,” Journal of the Association for Computing Machinery, vol. 68, no. 6, Sep. 2021. find
  132. F. Bretz, W. Maurer, W. Brannath, and M. Posch, “A graphical approach to sequentially rejective multiple test procedures,” Statistics in Medicine, vol. 28, no. 4, pp. 586–604, 2009. Figure 19: Object detection with simultaneous distribution-free guarantees on the expected intersection-over-union, re… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/angelopoulos-2021-gentle-introduction-conformal-prediction-distribution/