Elamur Research⌕
☰
home / papers / 2022 / Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast

Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast

Paper · 2022 · Kaifeng Bi, Lingxi Xie, Hengheng Zhang et al.

Summary

Numerical weather prediction (NWP) solves physical equations on supercomputers. It is slow, which limits timeliness and ensemble size, and its parameterizations introduce errors. Earlier deep learning forecasters such as FourCastNet were fast but less accurate than ECMWF's operational IFS.

Pangu-Weather is a data-driven global forecaster trained on 43 years of hourly ERA5 reanalysis at 0.25° resolution. It predicts five upper-air variables on 13 pressure levels plus four surface variables. It relies on two ideas. First, the 3D Earth Specific Transformer (3DEST) treats pressure levels as an extra spatial dimension, so the network can model relations between heights. Second, hierarchical temporal aggregation combines models with different lead times, which reduces the number of autoregressive steps and limits error accumulation.

The authors report that Pangu-Weather beats operational IFS on latitude-weighted RMSE and ACC for every variable and lead time from one hour to one week. It does this at a tiny fraction of the compute cost. This makes fast, large-member ensembles and extreme-event forecasting practical.

Key points

  • Data split: ERA5 1979–2017 for training, 2019 for validation, and 2018, 2020 and 2021 for testing.
  • Model size and training: several models totalling about 256M parameters, with separate models for 1 h, 3 h, 6 h and 24 h lead times. Each was trained for 100 epochs (around 15 days) on 192 NVIDIA Tesla-V100 GPUs. Training had still not fully converged.
  • Accuracy: 5-day Z500 RMSE of 296.7 for a single member, compared with 333.7 for operational IFS and 462.5 for a 100-member FourCastNet ensemble.
  • Speed: inference takes about 1.4 s on one GPU, which the authors report as over 10,000× faster than IFS.
  • Cyclone tracking: for Typhoon Yutu (2018), Pangu-Weather predicted the correct path 48 hours earlier than ECMWF-HRES.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Engineering · 133 worksEnvironmental Science · 9 worksEnvironmental Engineering · 3 worksEarth and Planetary Sciences · 3 worksBuilding and Construction · 5 worksAtmospheric Science · 1 work

Cite

Cite as Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu and Qi Tian. 2022. Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast. arXiv:2211.02556.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2211.02556 · arXiv 2211.02556

@misc{bi2022pangu,
  title = {{Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast}},
  author = {Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu and Qi Tian},
  year = {2022},
  doi = {10.48550/arxiv.2211.02556},
  eprint = {2211.02556},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.02556}
}

References

58 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Bauer, A. Thorpe, and G. Brunet, “The quiet revolution of numerical weather prediction,” Nature, vol. 525, no. 7567, pp. 47– 55, 2015. find
  2. W. C. Skamarock, J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, W. Wang, and J. G. Powers, “A description of the advanced research wrf version 2,” National Center For Atmospheric Research Boulder Co Mesoscale and Microscale . . . , Tech. Rep., 2005. find
  3. F. Molteni, R. Buizza, T. N. Palmer, and T. Petroliagis, “The ecmwf ensemble prediction system: Methodology and validation,” Quarterly journal of the royal meteorological society, vol. 122, no. 529, pp. 73–119, 1996. find
  4. H. Ritchie, C. Temperton, A. Simmons, M. Hortal, T. Davies, D. Dent, and M. Hamrud, “Implementation of the semi-lagrangian method in a high-resolution version of the ecmwf forecast model,” Monthly Weather Review, vol. 123, no. 2, pp. 489–514, 1995. find
  5. P. Bauer, T. Quintino, N. Wedi, A. Bonanni, M. Chrust, W. Deconinck, M. Diamantakis, P. Düben, S. English, J. Flemming et al., The ecmwf scalability programme: Progress and plans. European Centre for Medium Range Weather Forecasts, 2020. find
  6. T. Palmer, G. Shutts, R. Hagedorn, F. Doblas-Reyes, T. Jung, and M. Leutbecher, “Representing model uncertainty in weather and climate prediction,” Annual Review of Earth and Planetary Sciences, vol. 33, no. 1, pp. 163–193, 2005. find
  7. M. R. Allen, J. Kettleborough, and D. Stainforth, “Model error in weather and climate forecasting,” in ECMWF Predictability of Weather and Climate Seminar. European Centre for Medium Range Weather Forecasts, Reading, UK, 2002, pp. 279–304. find
  8. M. G. Schultz, C. Betancourt, B. Gong, F. Kleinert, M. Langguth, L. H. Leufen, A. Mozaffari, and S. Stadtler, “Can deep learning beat numerical weather prediction?” Philosophical Transactions of the Royal Society A, vol. 379, no. 2194, p. 20200097, 2021. find
  9. S. Scher and G. Messori, “Weather and climate forecasting with neural networks: using general circulation models (gcms) with different complexity as a study ground,” Geoscientific Model Development, vol. 12, no. 7, pp. 2797–2809, 2019. find
  10. S. Rasp, P. D. Dueben, S. Scher, J. A. Weyn, S. Mouatadid, and N. Thuerey, “Weatherbench: a benchmark data set for data-driven weather forecasting,” Journal of Advances in Modeling Earth Systems, vol. 12, no. 11, p. e2020MS002203, 2020. find
  11. J. A. Weyn, D. R. Durran, and R. Caruana, “Can machines learn to predict weather? using deep learning to predict gridded 500-hpa geopotential height from historical weather data,” Journal of Advances in Modeling Earth Systems, vol. 11, no. 8, pp. 2680–2693, 2019. find
  12. J. A. Weyn, D. R. Durran, R. Caruana, and N. Cresswell-Clay, “Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models,” Journal of Advances in Modeling Earth Systems, vol. 13, no. 7, p. e2021MS002502, 2021. find
  13. R. Keisler, “Forecasting global weather with graph neural networks,” arXiv preprint arXiv:2202.07575, 2022. — [arXiv:2202.07575](https://arxiv.org/abs/2202.07575) find
  14. J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli et al., “Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators,” arXiv preprint arXiv:2202.11214, 2022. — [arXiv:2202.11214](https://arxiv.org/abs/2202.11214) find
  15. Y. Hu, L. Chen, Z. Wang, and H. Li, “Swinvrnn: A data-driven ensemble forecasting model via learned distribution perturbation,” arXiv preprint arXiv:2205.13158, 2022. — [arXiv:2205.13158](https://arxiv.org/abs/2205.13158) find
  16. P. Bougeault, Z. Toth, C. Bishop, B. Brown, D. Burridge, D. H. Chen, B. Ebert, M. Fuentes, T. M. Hamill, K. Mylne et al., “The thorpex interactive grand global ensemble,” Bulletin of the American Meteorological Society, vol. 91, no. 8, pp. 1059–1072, 2010. find
  17. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738.
  18. H. Bao, L. Dong, and F. Wei, “Beit: Bert pre-training of image transformers,” arXiv preprint arXiv:2106.08254, 2021. — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) find
  19. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  20. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  21. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763.
  22. A. K. Betts, D. Z. Chan, and R. L. Desjardins, “Near-surface biases in era5 over the canadian prairies,” Frontiers in Environmental Science, vol. 7, p. 129, 2019. find
  23. Q. Jiang, W. Li, Z. Fan, X. He, W. Sun, S. Chen, J. Wen, J. Gao, and J. Wang, “Evaluation of the era5 reanalysis precipitation dataset over chinese mainland,” Journal of hydrology, vol. 595, p. 125660, 2021. find
  24. H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers et al., “The era5 global reanalysis,” Quarterly Journal of the Royal Meteorological Society, vol. 146, no. 730, pp. 1999–2049, 2020. find
  25. P. Lynch, “The origins of computer weather prediction and climate modeling,” Journal of computational physics, vol. 227, no. 7, pp. 3431–3444, 2008. find
  26. E. Kalnay, Atmospheric modeling, data assimilation and predictability. Cambridge university press, 2003. find
  27. D. J. Stensrud, Parameterization schemes: keys to understanding numerical weather prediction models. Cambridge University Press, 2009. find
  28. D. Randall, M. Khairoutdinov, A. Arakawa, and W. Grabowski, “Breaking the cloud parameterization deadlock,” Bulletin of the American Meteorological Society, vol. 84, no. 11, pp. 1547–1564, 2003. find
  29. R. Pincus, H. W. Barker, and J.-J. Morcrette, “A fast, flexible, approximate technique for computing radiative transfer in inhomogeneous cloud fields,” Journal of Geophysical Research: Atmospheres, vol. 108, no. D13, 2003. find
  30. A. Arakawa, “The cumulus parameterization problem: Past, present, and future,” Journal of climate, vol. 17, no. 13, pp. 2493– 2525, 2004. find
  31. A. Betts, “Non-precipitating cumulus convection and its parameterization,” Quarterly Journal of the Royal Meteorological Society, vol. 99, no. 419, pp. 178–196, 1973. find
  32. H.-L. Kuo, “Further studies of the parameterization of the influence of cumulus convection on large-scale flow,” Journal of Atmospheric Sciences, vol. 31, no. 5, pp. 1232–1240, 1974. find
  33. T. Nakaegawa, “High-performance computing in meteorology under a context of an era of graphical processing units,” Computers, vol. 11, no. 7, p. 114, 2022. find
  34. H. Olafsson and J.-W. Bao, Uncertainties in Numerical Weather Prediction. Elsevier, 2020. find
  35. I. M. Navon, “Data assimilation for numerical weather prediction: a review,” Data assimilation for atmospheric, oceanic and hydrologic applications, pp. 21–65, 2009. find
  36. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. find
  37. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, 2012.
  38. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Computer Vision and Pattern Recognition, 2016.
  39. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  40. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  41. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10 012–10 022.
  42. X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” Advances in neural information processing systems, vol. 28, 2015. find
  43. X. Shi, Z. Gao, L. Lausen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. Woo, “Deep learning for precipitation nowcasting: A benchmark and a new model,” Advances in neural information processing systems, vol. 30, 2017. find
  44. S. Agrawal, L. Barrington, C. Bromberg, J. Burge, C. Gazen, and J. Hickey, “Machine learning for precipitation nowcasting from radar images,” arXiv preprint arXiv:1912.12132, 2019. — [arXiv:1912.12132](https://arxiv.org/abs/1912.12132) find
  45. S. Ravuri, K. Lenc, M. Willson, D. Kangin, R. Lam, P. Mirowski, M. Fitzsimons, M. Athanassiadou, S. Kashem, S. Madge et al., “Skilful precipitation nowcasting using deep generative models of radar,” Nature, vol. 597, no. 7878, pp. 672–677, 2021. find
  46. V. Lebedev, V. Ivashkin, I. Rudenko, A. Ganshin, A. Molchanov, S. Ovcharenko, R. Grokhovetskiy, I. Bushmarinov, and D. Solo-mentsev, “Precipitation nowcasting with satellite imagery,” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2680–… find
  47. C. K. Sønderby, L. Espeholt, J. Heek, M. Dehghani, A. Oliver, T. Salimans, S. Agrawal, J. Hickey, and N. Kalchbrenner, “Met-net: A neural weather model for precipitation forecasting,” arXiv preprint arXiv:2003.12140, 2020. — [arXiv:2003.12140](https://arxiv.org/abs/2003.12140) find
  48. H. Hersbach, B. Bell, P. Berrisford, G. Biavati, A. Horányi, J. Muñoz Sabater, J. Nicolas, C. Peubey, R. Radu, I. Rozum et al., “Era5 hourly data on pressure levels from 1979 to present,” Copernicus climate change service (c3s) climate data store (cds), vol. 10, 2018. find
  49. ——, “Era5 hourly data on single levels from 1979 to present,” Copernicus Climate Change Service (C3S) Climate Data Store (CDS), vol. 10, 2018. find
  50. D. Richardson, J. Bidlot, L. Ferranti, A. Ghelli, C. Gibert, T. Hewson, M. Janousek, F. Prates, and F. Vitart, Verification statistics and evaluations of ECMWF forecasts in 2008-2009. ECMWF Reading, UK, 2009. find
  51. B. Fildier, W. D. Collins, and C. Muller, “Distortions of the rain distribution with warming, with and without self-aggregation,” Journal of Advances in Modeling Earth Systems, vol. 13, no. 2, p. e2020MS002256, 2021. find
  52. S. Bourdin, S. Fromang, W. Dulac, J. Cattiaux, and F. Chauvin, “Intercomparison of four tropical cyclones detection algorithms on era5,” EGUsphere, pp. 1–43, 2022. find
  53. P. Malakar, A. Kesarkar, J. Bhate, V. Singh, and A. Deshamukhya, “Comparison of reanalysis data sets to comprehend the evolution of tropical cyclones over north indian ocean,” Earth and Space Science, vol. 7, no. 2, p. e2019EA000978, 2020. find
  54. L. Magnusson, S. Majumdar, R. Emerton, D. Richardson, M. Alonso-Balmaseda, C. Baugh, P. Bechtold, J.-R. Bidlot, A. Bonanni, M. Bonavita, N. Bormann, A. Brown, P. Browne, H. Carr, M. Dahoui, G. D. Chiara, M. Diamantakis, D. Duncan, S. English, R. Forbes, A. J. Geer, T. Haiden, S. Healy, T. Hewson, B.… find
  55. P. White, “Newsletter no. 102 - winter 2004/05,” 01 2005. [Online]. Available: https://www.ecmwf.int/node/14623 ↗
  56. K. R. Knapp, M. C. Kruk, D. H. Levinson, H. J. Diamond, and C. J. Neumann, “The international best track archive for climate stewardship (ibtracs) unifying tropical cyclone data,” Bulletin of the American Meteorological Society, vol. 91, no. 3, pp. 363–376, 2010. find
  57. K. R. Knapp, H. J. Diamond, J. P. Kossin, M. C. Kruk, C. Schreck et al., “International best track archive for climate stewardship (ibtracs) project, version 4,” NOAA National Centers for Environmental Information, 2018. find
  58. E. P. Diaconescu and R. Laprise, “Singular vectors in atmospheric sciences: A review,” Earth-science reviews, vol. 113, no. 3-4, pp. 161– 175, 2012. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bi-2022-pangu-weather-3d-high/