Elamur Research
☰
home / papers / 2019 / Invariant Risk Minimization

Invariant Risk Minimization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1907.02893

Summary

Machine learning models minimize error on the training data, so they absorb every correlation indiscriminately, including spurious ones that arose from biases in data collection. The classic example: cows are mostly photographed on green grass and camels in the desert, so the model learns to recognize the background instead of the animal. The authors propose Invariant Risk Minimization (IRM). This principle requires the data to be collected in several separate "environments," that is, under different conditions. The model looks for a data representation for which the same classifier is optimal in every environment. This preserves stable, causal relationships and discards unstable ones. Using a simple linear model, the authors show that ERM, robust optimization, domain adaptation and invariant causal prediction fail to find the invariant predictor. They also formally connect invariance with causality and with out-of-distribution (OOD) generalization. On Colored MNIST, the practical variant IRMv1 ignores the misleading color feature in a setting where ERM "breaks" on it. The paper has become one of the foundational works in research on OOD generalization.

Key points

  • Toy example with structural equations: only the regression of Y on the causal variable X1 has coefficients that stay the same across environments. Only this regression has finite OOD risk. Regressions that use the effect X2 depend on the environment.
  • Robust learning (Proposition 2): under certain conditions it reduces to minimizing a weighted sum of the per-environment risks. As a result, like ERM, it assigns a large weight to the spurious feature X2.
  • IRMv1: a penalty is added to standard ERM. The penalty is the squared norm of the gradient of each environment's risk with respect to a fixed "dummy" scalar classifier w = 1.0, and its weight is set by the hyperparameter λ.
  • Colored MNIST: color correlates with the label differently in the two training environments, and at test time this correlation flips sign. At test time ERM reaches about 17% accuracy and IRM about 67%, close to the oracle trained on grayscale images (≈73%). These numbers are not in the provided excerpt, because the experiments section was omitted; I quoted them from memory of the paper, so they should be checked against the original before publication.
  • Limitations and directions: if the model has too little capacity, the invariant predictor may turn out to be trivial, in which case it is better to fall back to ERM. The paper closes with a Socratic dialogue offering ideas for further research: RL episodes as environments, fairness when groups of people are treated as environments, and a connection to group theory.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

98 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1907.02893 · arXiv 1907.02893

@misc{arjovsky2019invariant,
  title = {{Invariant Risk Minimization}},
  author = {Arjovsky, Martin and Bottou, Léon and Gulrajani, Ishaan and Lopez-Paz, David},
  year = {2019},
  doi = {10.48550/arxiv.1907.02893},
  eprint = {1907.02893},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1907.02893}
}

References

56 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Aldrich. Autonomy. Oxford Economic Papers, 1989. find
  2. James Andrew Bagnell. Robust supervised learning. In AAAI, 2005. find
  3. Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign Overfitting in Linear Regression. arXiv, 2019. find
  4. Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018. find
  5. Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NIPS. 2007. find
  6. Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization. Princeton University Press, 2009. find
  7. Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv, 2019. find
  8. Léon Bottou, Corinna Cortes, John S. Denker, Harris Drucker, Isabelle Guyon, Lawrence D. Jackel, Yann Le Cun, Urs A. Muller, Eduard Säckinger, Patrice Simard, and Vladimir Vapnik. Comparison of classifier methods: a case study in handwritten digit recognition. In ICPR, 1994. find
  9. Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-features models works surprisingly well on imagenet. In ICLR, 2019. find
  10. Joan Bruna and Stephane Mallat. Invariant scattering convolution networks. TPAMI, 2013. find
  11. Joan Bruna, Stephane Mallat, Emmanuel Bacry, and Jean-Franois Muzy. Intermittent process analysis with scattering moments. The Annals of Statistics, 2015. find
  12. Nancy Cartwright. Two theorems on invariance and causality. Philosophy of Science, 2003. find
  13. Patricia W. Cheng and Hongjing Lu. Causal invariance as an essential constraint for creating a causal representation of the world. The Oxford handbook of causal reasoning, 2017. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. find
  15. John Duchi, Peter Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. arXiv, 2016. find
  16. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016. find
  17. Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR, 2019. find
  18. AmirEmad Ghassami, Saber Salehkaleybar, Negar Kiyavash, and Kun Zhang. Learning causal structures using regression invariance. In NIPS, 2017. find
  19. Patrick J. Grother. NIST Special Database 19: Handprinted forms and characters database. https://www.nist.gov/srd/nist-special-database-19, 1995. File doc/doc.ps in the 1995 NIST CD ROM NIST Special Database 19. ↗
  20. Trygve Haavelmo. The probability approach in econometrics. Econometrica: Journal of the Econometric Society, 1944. find
  21. Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. arXiv, 2017. find
  22. Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 2018. find
  23. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. find
  24. Fredrik D. Johansson, David A. Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant representations. AISTATS, 2019. find
  25. Niki Kilbertus, Giambattista Parascandolo, and Bernhard Schölkopf. Generalization in anti-causal learning. arXiv, 2018. find
  26. Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown environments. In SIGKDD, 2018. find
  27. Brenden M. Lake, Tomer D. Ullman, Joshua B Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 2017. find
  28. James M. Lee. Introduction to Smooth Manifolds. Springer, 2003. find
  29. David Lewis. Counterfactuals. John Wiley & Sons, 2013. find
  30. Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018. find
  31. David Lopez-Paz. From dependence to causation. PhD thesis, University of Cambridge, 2016. find
  32. David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Discovering causal signals in images. In CVPR, 2017. find
  33. Gilles Louppe, Michael Kagan, and Kyle Cranmer. Learning to pivot with adversarial networks. In Advances in neural information processing systems, pages 981–990, 2017. find
  34. Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In NIPS, 2018. find
  35. Gary Marcus. Deep learning: A critical appraisal. arXiv, 2018. find
  36. Nicolai Meinshausen. Causality from a distributional robustness point of view. In Data Science Workshop (DSW), 2018. find
  37. Nicolai Meinshausen and Peter Bühlmann. Maximin effects in inhomogeneous large-scale data. The Annals of Statistics, 2015. find
  38. Sandra D. Mitchell. Dimensions of scientific law. Philosophy of Science, 2000. find
  39. Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009. find
  40. Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. JRSS B, 2016. find
  41. Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017. find
  42. Michael Redhead. Incompleteness, non locality and realism. a prolegomenon to the philosophy of quantum mechanics. 1987. find
  43. Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 2018. find
  44. Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 1974. find
  45. Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In ICML, 2012. find
  46. Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. ICLR, 2018. find
  47. Brian Skyrms. Causal necessity: a pragmatic investigation of the necessity of laws. Yale University Press, 1980. find
  48. Bob L. Sturm. A simple method to determine if a music information retrieval system is a “horse”. IEEE Transactions on Multimedia, 2014. find
  49. Antonio Torralba and Alexei Efros. Unbiased look at dataset bias. In CVPR, 2011. find
  50. Vladimir Vapnik. Principles of risk minimization for learning theory. In NIPS. 1992. find
  51. Vladimir N. Vapnik. Statistical Learning Theory. John Wiley & Sons, 1998. find
  52. Max Welling. Do we still need models or just more data and compute?, 2019. find
  53. Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In NIPS. 2017. find
  54. James Woodward. Making things happen: A theory of causal explanation. Oxford university press, 2005. find
  55. Sewall Wright. Correlation and causation. Journal of agricultural research, 1921. find
  56. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2016. find