Elamur Research⌕
☰
home / papers / 2018 / Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking

Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking

Paper · 2018 · Zhe Jia, Marco Maggioni, Benjamin Staiger et al.

Summary

This technical report from the High Performance Computing R&D Team at Citadel (Chicago) examines NVIDIA's Volta GPU architecture. Much of the hardware's low-level behaviour is not publicly documented, so the authors use microbenchmarking: small, targeted test programs that expose hidden properties of the chip. Based on its table of contents, the study covers four areas. The first is instruction-level details: control information, the scheduler and instruction encoding. The second is the full memory hierarchy, from register banks to TLBs. The third is the latency and throughput of native instructions, atomics and tensor cores. The fourth is Volta boards with NVLink. An opening chapter explains why such details matter to performance-minded programmers. The supplied extract contains only the front matter and table of contents, so measured values and final conclusions cannot be summarised here. The report matters because it gives developers of highly tuned GPU code an empirical map of Volta's internals that vendor documentation does not provide.

Key points

  • The authors are Zhe Jia, Marco Maggioni, Benjamin Staiger and Daniele P. Scarpazza (Citadel). The report is arXiv:1804.06826 (cs.DC), first edition, dated April 18, 2018. Zhe Jia previewed parts of it at the 2018 GPU Technology Conference in San Jose.
  • The instruction chapter covers control information embedded in instructions, scheduler behaviour and the encoding of instructions.
  • The memory-hierarchy chapter has eight parts:
    • L1 data cache
    • L2 cache
    • instruction caches
    • constant memory and cache
    • register banks
    • shared memory
    • global memory
    • TLBs
  • The performance chapter measures native instructions, atomic operations, tensor cores and overall floating-point performance.
  • A separate chapter studies NVLink-enabled Volta boards. It covers peer-to-peer and host communication, arithmetic throughput, memory bandwidth and latency.

Interesting ideas

  • I1. Reverse-engineering through microbenchmarks. Narrowly scoped probes can reveal properties a vendor does not document. The same approach could characterise any opaque platform, such as cloud instances, accelerators or storage systems, before tuning workloads for it.
  • I2. Layer-by-layer memory study. Measuring each cache level, register banks and TLBs separately shows where data-access bottlenecks occur. This is useful for anyone optimising memory-bound code, including large NLP model inference.
  • I3. Separating latency from throughput. Reporting both for instructions, atomics and tensor cores helps decide whether code is limited by dependency chains or by issue capacity. This distinction applies broadly in performance engineering.
  • I4. Starting with "why it matters". Opening with motivation before the low-level measurements makes dense hardware findings easier to act on. The same structure works well for internal technical reports.
  • I5. Versioned, living report. Releasing numbered editions on arXiv lets findings grow as new hardware or insights appear. A shared research library could adopt this pattern for evolving internal documentation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institut de Recherche en Informatique de ToulouseToulouse · France36 works · 2,193 citations

Also on the byline: Office National d'Études et de Recherches Aérospatiales.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksHardware and Architecture · 167 worksComputer Graphics and Computer-Aided Design · 16 works

🧭 Topics 1Research subjects and languages the work is about.GPU & parallel computing · 41 works

Cite

Cite as Zhe Jia, Marco Maggioni, Benjamin Staiger and Daniele P. Scarpazza. 2018. Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking. arXiv technical report.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1804.06826

@misc{jia2018dissecting,
  title = {{Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking}},
  author = {Zhe Jia, Marco Maggioni, Benjamin Staiger and Daniele P. Scarpazza},
  year = {2018},
  doi = {10.4230/oasics.wcet.2024.3},
  eprint = {1804.06826},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.4230/oasics.wcet.2024.3}
}

References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Lindholm, J. Nickolls, S. Oberman, and J. Montrym, “NVIDIA Tesla: A unified graphics and computing architecture,” IEEE Micro, vol. 28, no. 2, pp. 39–55, Mar. 2008. [Online]. Available: http://dx.doi.org/10.1109/MM.2008.31 ↗
  2. NVIDIA’s Next Generation CUDA Compute Architecture: Fermi. Corporation, 2009. find
  3. Kepler GK110 Whitepaper. NVIDIA Corporation, 2012. find
  4. GeForce GTX 980 - Featuring Maxwell, The Most Advanced GPU Ever Made, Whitepaper. NVIDIA Corporation, 2014. find
  5. Pascal GP100 Whitepaper. NVIDIA Corporation, 2016. find
  6. Nvidia Tesla V100 GPU Architecture, The World’s Most Advanced Data Center GPU. NVIDIA Corporation, 2017.
  7. Y. Hou, “Asfermi,” 2011. [Online]. Available: https://github.com/hyqneuron/asfermi ↗
  8. C. Wang, Z. Jia, and K. Chen, “Tuning performance on Kepler GPUs: An introduction to Kepler assembler and its usage in CNN optimization,” in GPU Technology Conference Presentation, vol. 6173, 2015. find
  9. S. Gray, “maxas,” 2016. [Online]. Available: https://github.com/NervanaSystems/maxas/wiki/Control-Codes ↗
  10. H. Wong, M. M. Papadopoulou, M. Sadooghi-Alvandi, and A. Moshovos, “Demystifying GPU microarchitecture through microbenchmarking,” in 2010 IEEE International Symposium on Performance Analysis of Systems Software (ISPASS), March 2010, pp. 235–246. find
  11. X. Zhang, G. Tan, S. Xue, J. Li, K. Zhou, and M. Chen, “Understanding the GPU microarchitecture to achieve bare-metal performance tuning,” in Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and NVIDIA Practice of Parallel Programming, ser. PPoPP ’17. ACM, 2017, pp. 31–43. find
  12. X. Mei and X. Chu, “Dissecting GPU memory hierarchy through microbenchmarking,” IEEE Transactions on Parallel and Distributed Systems, vol. 28, no. 1, pp. 72–86, Jan 2017. find
  13. A. Lavin, “maxDNN: an efficient convolution kernel for deep learning with Maxwell GPUs,” arXiv preprint arXiv:1501.06633, 2015. find
  14. G. Tan, L. Li, S. Triechle, E. Phillips, Y. Bao, and N. Sun, “Fast implementation of DGEMM on Fermi GPU,” in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 2011, p. 35. find
  15. J. Kurzak, S. Tomov, and J. Dongarra, “Autotuning GEMM kernels for the Fermi GPU,” IEEE Transactions on Parallel and Distributed Systems, vol. 23, no. 11, pp. 2045–2057, 2012. find
  16. R. Nath, S. Tomov, and J. Dongarra, “An improved MAGMA GEMM for Fermi graphics processing units,” The International Journal of High Performance Computing Applications, vol. 24, no. 4, pp. 511–515, 2010. find
  17. J. Lai and A. Seznec, “Performance upper bound analysis and optimization of SGEMM on Fermi and Kepler GPUs,” in Code Generation and Optimization (CGO), 2013 IEEE/ACM International Symposium on. IEEE, 2013, pp. 1–10. find
  18. “Nvidia cuobjdump and nvdisasm,” 2016. [Online]. Available: https://docs.nvidia.com/cuda/cuda-binary-utilities/ ↗
  19. Nvidia Tesla V100 GPU Accelerator. BIBLIOGRAPHY New York, NY, USA: NVIDIA Corporation, 2017. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jia-2018-dissecting-nvidia-volta-gpu-architecture/