Elamur Research⌕
☰
home / papers / 2021 / Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction

Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction

Paper · 2021 · Cheng Sun, Min Sun and Hwann-Tzong Chen

Summary

NeRF-style novel view synthesis gives high-quality renderings from posed photographs, but fitting a single scene usually takes hours or days of GPU time. This paper (DVGO) shows that a radiance field can be learned from scratch in about a quarter of an hour on one GPU while reaching NeRF-level quality. Instead of hiding geometry inside an MLP, the authors store volume density directly in a dense voxel grid. Color comes from a second grid of features, decoded by a small MLP. Two ingredients make this explicit approach work. First, raw grid values are interpolated trilinearly and only then passed through the activation, which lets a single cell express a sharp surface, so a coarse grid is enough. Second, the optimization is regularized so it does not fill empty space with "cloud": densities start near zero, and voxels seen by few cameras get smaller learning rates. Training proceeds from coarse geometry to fine detail. On five inward-facing benchmarks the method matches or beats NeRF, making per-scene reconstruction practical.

Key points

  • Training time drops from about 10–20 hours (NeRF) to about 15 minutes on a single NVIDIA RTX 2080 Ti, roughly two orders of magnitude faster. Rendering is about 45× faster than NeRF.
  • On the Lego scene the model reaches 23.64 PSNR after 2.33 min, 32.72 after 5.07 min and 34.22 after 13.72 min.
  • A density grid of about 160³ voxels is enough. Earlier voxel-based methods needed 512³–1300³ for comparable quality.
  • No cross-scene pretraining, external MVS depth, or conversion from a pre-trained NeRF is required. Each scene is optimized directly.
  • The pipeline has four parts:
    • automatic search for a tight bounding box;
    • free-space skipping;
    • a coarse geometry stage;
    • a fine stage for detail and view-dependent effects.
  • Code is released as DirectVoxGO.

Interesting ideas

  • I1. Activate after interpolating, not before. Interpolating raw (pre-activation) values and applying the nonlinearity afterwards gives much sharper boundaries than interpolating already-activated outputs. This trick applies to any discretized field or lookup table that feeds a nonlinearity.
  • I2. Scale learning rates by evidence. Parameters supported by few observations (here, voxels visible from few views) receive smaller learning rates, so they do not overfit a handful of samples. The same principle could help any sparse, explicitly parameterized model where coverage is uneven.
  • I3. Initialize to "empty". Starting densities near zero keeps the solution from being biased toward regions close to the cameras. Choosing a neutral initial state is a cheap way to steer an explicit optimization away from degenerate solutions.
  • I4. Explicit structure where it matters, small networks elsewhere. A plain grid handles geometry for speed, while a feature grid plus a shallow MLP handles the harder appearance modeling. This hybrid balances speed and expressiveness without a deep network.
  • I5. Coarse-to-fine with a tight bounding box. First locating the region of interest and allocating resolution only inside it saves memory and compute. This design generalizes to many reconstruction or search problems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Tsing Hua UniversityHsinchu · Taiwan10 works · 1,850 citations🏢AsusTaipei · Taiwan3 works · 2,674 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksComputer Graphics and Computer-Aided Design · 16 worksComputational Mechanics · 5 works

Cite

Cite as Cheng Sun, Min Sun and Hwann-Tzong Chen. 2021. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. arXiv:2111.11215.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2111.11215 · arXiv 2111.11215

@misc{sun2021direct,
  title = {{Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction}},
  author = {Cheng Sun, Min Sun and Hwann-Tzong Chen},
  year = {2021},
  doi = {10.48550/arxiv.2111.11215},
  eprint = {2111.11215},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.11215}
}

References

72 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021. 1, 3, 4, 6, 7, 9, 11, 12, 13 find
  2. Sai Bi, Zexiang Xu, Pratul P. Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milos Hasan, Yannick Hold-Geoffroy, David J. Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arxiv CS.CV 2106.01970, 2020. 2 find
  3. Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P. A. Lensch. Nerd: Neural reflectance decomposition from image collections. In ICCV, 2021. 2 find
  4. Chris Buehler, Michael Bosse, Leonard McMillan, Steven J. Gortler, and Michael F. Cohen. Unstructured lumigraph rendering. In SIGGRAPH, 2001. 2 find
  5. Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021. 2 find
  6. Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021. 1, 3, 7 find
  7. Abe Davis, Marc Levoy, and Frédo Durand. Unstructured light fields. Comput. Graph. Forum, 2012. 2 find
  8. Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In SIGGRAPH, 1996. 2 find
  9. Boyang Deng, Jonathan T. Barron, and Pratul P. Srinivasan. JaxNeRF: an efficient JAX implementation of NeRF, 2020. 6, 7, 11, 13, 17 find
  10. Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. arxiv CS.CV 2107.02791, 2021. 1, 3 find
  11. Helisa Dhamo, Keisuke Tateno, Iro Laina, Nassir Navab, and Federico Tombari. Peeking behind objects: Layered depth prediction from a single image. Pattern Recognit. Lett., 2019. 2 find
  12. John Flynn, Michael Broxton, Paul E. Debevec, Matthew DuVall, Graham Fyffe, Ryan S. Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In CVPR, 2019. 2 find
  13. Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, 2021. 2 find
  14. Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021. 2 find
  15. Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien P. C. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In ICCV, 2021. 1, 2, 3, 7, 13 find
  16. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. The lumigraph. In SIGGRAPH, 1996. 2 find
  17. Tong He, John P. Collomosse, Hailin Jin, and Stefano Soatto. Deepvoxels++: Enhancing the fidelity of novel view synthesis from 3d voxel embeddings. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomás Pajdla, and Jianbo Shi, editors, ACCV, 2020. 2, 18 find
  18. Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul E. Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV, 2021. 1, 2, 3, 5, 7, 10, 13 find
  19. Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In ICCV, 2021. 2 find
  20. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6
  21. Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph., 2017. 6, 11, 12, 17 find
  22. Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In ICML, 2021. 2 find
  23. Anat Levin and Frédo Durand. Linear view synthesis using a dimensionality gap light field prior. In CVPR, 2010. 2 find
  24. Marc Levoy and Pat Hanrahan. Light field rendering. In SIGGRAPH, 1996. 2 find
  25. Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021. 2 find
  26. Zhengqi Li, Wenqi Xian, Abe Davis, and Noah Snavely. Crowdsampling the plenoptic function. In ECCV, 2020. 2 find
  27. Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: bundle-adjusting neural radiance fields. In ICCV, 2021. 2 find
  28. Yen-Chen Lin, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In IROS, 2021. 2 find
  29. David B. Lindell, Julien N. P. Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In CVPR, 2021. 1, 7, 13 find
  30. Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurIPS, 2020. 1, 2, 3, 5, 6, 7, 11, 12, 13, 15, 16, 17 find
  31. Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. arxiv CS.CV 2107.13421, 2021. 1, 3, 7, 10 find
  32. Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Trans. Graph., 2019. 2, 7, 13, 15, 16, 17, 18 find
  33. Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021. 2 find
  34. Nelson L. Max. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph., 1995. 3 find
  35. Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. Gnerf: Gan-based neural radiance field without posed camera. In ICCV, 2021. 2 find
  36. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph., 2019. 2, 18 find
  37. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 1, 2, 3, 5, 6, 7, 9, 11, 12, 13, 14, 15, 16, 17, 18 find
  38. Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In ICCV, 2021. 2 find
  39. Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, 2021. 9 find
  40. Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Deformable neural radiance fields. In ICCV, 2021. 2 find
  41. Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arxiv CS.CV 2106.13228, 2021. 2 find
  42. Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021. 2 find
  43. Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In CVPR, 2021. 1 find
  44. Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In ICCV, 2021. 1, 3, 7, 13, 15, 16, 17 find
  45. Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: generative radiance fields for 3d-aware image synthesis. In NeurIPS, 2020. 2 find
  46. Jonathan Shade, Steven J. Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In SIGGRAPH, 1998. 2 find
  47. Lixin Shi, Haitham Hassanieh, Abe Davis, Dina Katabi, and Frédo Durand. Light field reconstruction using sparsity in the continuous fourier domain. ACM Trans. Graph., 2014. 2 find
  48. Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context-aware layered depth inpainting. In CVPR, 2020. 2 find
  49. Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In CVPR, 2019. 2, 6, 11, 12, 18 find
  50. Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In NeurIPS, 2019. 7, 13, 15, 16, 17, 18 find
  51. Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021. 2 find
  52. Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In CVPR, 2019. 2 find
  53. Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In CVPR, 2021. 2 find
  54. Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020. 3 find
  55. Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph., 2019. 2 find
  56. Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a deforming scene from monocular video. In ICCV, 2021. 2 find
  57. Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In CVPR, 2020. 2 find
  58. Shubham Tulsiani, Richard Tucker, and Noah Snavely. Layer-structured 3d scene inference via view synthesis. In ECCV, 2018. 2 find
  59. Michael Waechter, Nils Moehrle, and Michael Goesele. Let there be color! large-scale texturing of 3d reconstructions. In ECCV, 2014. 2 find
  60. Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas A. Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021. 1, 3, 7, 18 find
  61. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004. 6 find
  62. Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arxiv CS.CV 2102.07064, 2021. 2 find
  63. Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. Nex: Real-time view synthesis with neural basis expansion. In CVPR, 2021. 2, 3, 10 find
  64. Daniel N. Wood, Daniel I. Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David Salesin, and Werner Stuetzle. Surface light fields for 3d photography. In SIGGRAPH, 2000. 2 find
  65. Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021. 2 find
  66. Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In CVPR, 2020. 6, 11, 12, 16 find
  67. Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In ICCV, 2021. 1, 2, 3, 5, 7, 10, 11, 13, 17 find
  68. Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR, 2021. 3 find
  69. Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arxiv CS.CV 2010.07492, 2020. 3 find
  70. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 6 find
  71. Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul E. Debevec, William T. Freeman, and Jonathan T. Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. arxiv CS.CV 2106.01970, 2021. 2 find
  72. Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph., 2018. 2 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2021-direct-voxel-grid-optimization-super/