Elamur Research⌕
☰
home / papers / 2019 / Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Paper · 2019 · Ang Li, Shuaiwen Leon Song, Jieyang Chen et al.

Summary

Multi-GPU computing is now essential for deep learning, big data and large-scale simulation. However, practitioners have little solid knowledge of how GPUs are wired together inside and across servers, or how much that wiring affects real workloads. The authors run a systematic empirical study of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NV-SLI, NVSwitch and GPUDirect-enabled InfiniBand. They test six platforms: P100-DGX-1, V100-DGX-1, DGX-2, the OLCF SummitDev and Summit supercomputers, and a two-GPU RTX-2080 SLI machine. For both peer-to-peer (P2P) and collective (CL) communication, they measure startup latency, sustained one- and two-way bandwidth, topology, routing, efficiency and NUMA behaviour. The central finding is four previously undocumented NUMA effects in GPU networks. Three stem from NVLink topology, connectivity and routing, and one from a PCIe chipset design issue. As a result, the choice of GPUs an application runs on can noticeably change both communication speed and total runtime. The measurements are intended to support multi-GPU performance models for task allocation, scheduling and migration in shared AI clouds and HPC centres.

Key points

  • Six testbeds: DGX-1 with eight P100 GPUs (NVLink-V1), DGX-1 with eight V100 GPUs (NVLink-V2), DGX-2 with 16 V100 GPUs (NVSwitch), SummitDev (54 nodes × 4 P100), Summit (4600 nodes × 6 V100) and an SLI box with two RTX-2080 (Turing) cards.
  • P100-DGX-1 uses a "Hypercube Mesh" of NVLinks, so GPUs not directly linked (e.g., GPU-0 to GPU-7) see NUMA behaviour. NVLink does not route traffic on its own; transfers between such GPUs need an intermediate GPU chosen by the user.
  • NVLink-V2 changes:
    • Each GPU gets six link slots instead of four, and each link is about 25% faster.
    • V100-DGX-1 adds a backbone ring whose links carry double bandwidth.
  • DGX-2 contains 12 NVSwitch chips, six per baseboard. Each chip is an 18-port non-blocking crossbar, giving any GPU pair on one baseboard 300 GB/s through a single switch hop.
  • On SummitDev and Summit, each node splits into two CPU+GPU subnetworks with no direct GPU links between them.
    • The halves are joined by an X-Bus at 38 GB/s (SummitDev) or 64 GB/s (Summit).
    • The Turing SLI bridge provides 50 GB/s bidirectional between two GPUs.

Interesting ideas

  • I1. Interconnect as a NUMA system: Treating the GPU network as a NUMA system, rather than a flat pool, is a useful mental model. Schedulers for clusters or shared servers can rank device combinations by link distance before placing jobs.
  • I2. Separate P2P and collective tests: Benchmarking point-to-point and collective patterns separately shows that a topology can be strong for one and weak for the other. Any distributed-training setup could reuse this two-axis evaluation.
  • I3. One method across many platforms: Applying the same measurement procedure to several hardware generations and vendors' topologies makes differences directly comparable. This is a good template for internal hardware or cloud-instance comparisons.
  • I4. Routing through an intermediate GPU: Because NVLink lacks self-routing, explicitly choosing an intermediate GPU for indirect transfers is a practical tuning lever. Libraries that move data between devices can apply it when no direct link exists.
  • I5. Measurements as model inputs: Using empirical latency and bandwidth tables as inputs to performance models, instead of vendor peak figures, makes cost estimates for task migration and allocation far more realistic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Pacific Northwest National LaboratoryRichland · United States4 works · 1,537 citations🔬Oak Ridge National LaboratoryOak Ridge · United States6 works · 881 citations🎓William & MaryWilliamsburg · United States7 works · 794 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksHardware and Architecture · 167 worksComputer Networks and Communications · 100 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksGPU & parallel computing · 41 works

Cite

Cite as Ang Li, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan Tallent and Kevin Barker. 2019. Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect. arXiv:1903.04611.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/tpds.2019.2928289 · arXiv 1903.04611

@misc{li2019evaluating,
  title = {{Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect}},
  author = {Ang Li, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan Tallent and Kevin Barker},
  year = {2019},
  doi = {10.1109/tpds.2019.2928289},
  eprint = {1903.04611},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/tpds.2019.2928289}
}

References

60 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” arXiv preprint arXiv:1706.02677, 2017. find
  2. O. Fuhrer, T. Chadha, T. Hoefler, G. Kwasniewski, X. Lapillonne, D. Leutwyler, D. Lüthi, C. Osuna, C. Schär, T. C. Schulthess et al., “Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 gpus with cosmo 5.0,” Geoscientific Model Development, 2018. find
  3. H. Mikami, H. Suganuma, P. U-chupala, Y. Tanaka, and Y. Kageyama, “Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash,” arXiv preprint arXiv:1811.05233, 2018. find
  4. NVIDIA, “NVIDIA DGX-1 System Architecture White Paper,” 2017. find
  5. NVIDIA, “NVIDIA DGX-2H The World’s Most Powerful System for The Most Complex AI Challenges,” 2018. find
  6. OLCF, “Summit: The Next Leap in Leadership-Class Computing Systems for Open Science,” https://www.olcf.ornl.gov/for-users/system-user-guides/summit/. ↗
  7. OLCF, “Sierra: Supporting NNSA’s stockpile stewardship mission through simulation in lieu of underground testing,” https://computation.llnl.gov/computers/sierra. ↗
  8. S. Pabst, A. Koch, and W. Straßer, “Fast and scalable cpu/gpu collision detection for rigid and deformable surfaces,” in Computer Graphics Forum. Wiley Online Library, 2010. find
  9. Q. Xu, H. Jeon, and M. Annavaram, “Graph processing on GPU: Where are the bottlenecks?” in International Symposium on Workload Characterization (IISWC). IEEE, 2014. find
  10. “The System Bottleneck Shifts to PCI-Express,” https://www.nextplatform.com/2017/07/14/system-bottleneck-shifts-pci-express/. ↗
  11. D. Ziakas, A. Baum, R. A. Maddox, and R. J. Safranek, “Intel R quickpath interconnect architectural features supporting scalable system architectures,” in High Performance Interconnects (HOTI), 2010 IEEE 18th Annual Symposium on. IEEE, 2010. find
  12. D. Foley and J. Danskin, “Ultra-Performance Pascal GPU and NVLink Interconnect,” IEEE Micro, 2017. find
  13. NVIDIA, “SLI best practices,” Tech. Rep., 2007. find
  14. AMD, “ATI CrossFire Pro User Guide,” Tech. Rep., 2009. find
  15. G. Kim, M. Lee, J. Jeong, and J. Kim, “Multi-GPU system design with memory networks,” in 47th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2014. find
  16. NVIDIA, “NVIDIA NVSWITCH – The World’s Highest-Bandwidth On-Node Switch,” http://images.nvidia.com/content/pdf/nvswitch-technical-overview.pdf. ↗
  17. P. Grun, “Introduction to infiniband for end users,” White paper, InfiniBand Trade Association, 2010. find
  18. NVIDIA, “Developing a Linux Kernel Module using GPUDirect RDMA,” http://docs.nvidia.com/cuda/gpudirect-rdma/index. html. ↗
  19. AMD, “ROCm Driver RDMA Peer to Peer Support,” https://github.com/RadeonOpenCompute/ROCnRDMA. ↗
  20. NVIDIA, “CUDA SDK Code Samples,” 2015. find
  21. NVIDIA, “NCCL Tests,” https://github.com/NVIDIA/nccl-tests. ↗
  22. NVIDIA, “NVIDIA Collective Communications Library (NCCL-V1),” https://github.com/NVIDIA/nccl. ↗
  23. NVIDIA, “NVIDIA Collective Communications Library (NCCL-V2),” https://developer.nvidia.com/nccl. ↗
  24. “MPI-GPU-BW,” https://github.com/frroberts/MPI-GPU-BW. ↗
  25. A. Li, S. Song, J. Chen, X. Liu, N. Tallent, and K. Barker, “Tartan: Evaluating Modern GPU Interconnect via a Multi-GPU Benchmark Suite,” in International Symposium on Workload Characterization (IISWC). IEEE, 2018. find
  26. J. D. Little, “A proof for the queuing formula: L= λ W,” Operations research, 1961. find
  27. Cirrascale, “Cirrascale SR3514: Unexpected Performance Inequality. Technical Brief M901A-092014.” find
  28. AMD, “ROCm Communication Collectives Library (RCCL),” https://github.com/ROCmSoftwarePlatform/rccl. ↗
  29. OLCF, “Summit Early Access Development Platform,” https://www.olcf.ornl.gov/for-users/system-user-guides/summitdev-quickstart-guide/. ↗
  30. NVIDIA, “Developing a Linux Kernel Module using GPUDirect RDMA,” https://docs.nvidia.com/cuda/gpudirect-rdma/index. html. ↗
  31. S. Shams, R. Platania, K. Lee, and S.-J. Park, “Evaluation of Deep Learning Frameworks Over Different HPC Architectures,” in 37th International Conference on Distributed Computing Systems (ICDCS). IEEE, 2017. find
  32. N. R. Tallent, N. A. Gawande, C. Siegel, A. Vishnu, and A. Hoisie, “Evaluating On-Node GPU Interconnects for Deep Learning Workloads,” in International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems. Springer, 2017. find
  33. S. Potluri, A. Goswami, D. Rossetti, C. Newburn, M. G. Venkata, and N. Imam, “GPU-Centric Communication on NVIDIA GPU Clusters with InfiniBand: A Case Study with OpenSHMEM,” in HiPC-24. IEEE, 2017. find
  34. E. Agostini, D. Rossetti, and S. Potluri, “GPUDirect Async: Exploring GPU synchronous communication techniques for InfiniBand clusters,” Journal of Parallel and Distributed Computing, 2018. find
  35. M. LeBeane, K. Hamidouche, B. Benton, M. Breternitz, S. K. Reinhardt, and L. K. John, “GPU triggered networking for intra-kernel communications,” in International Conference for High Performance Computing, Networking, Storage and Analysis (SC). ACM, 2017. find
  36. J. Yin, “Early experiences with Machine Learning and Deep Learning on Summit/SummitDev,” https://www.olcf.ornl.gov/wp-content/uploads/2018/12/summit training mldl.pdf, 2018. ↗
  37. NERSC, “Perlmutter: NERSC’s Next Supercomputer,” https://www.nersc.gov/systems/perlmutter/. ↗
  38. A. Li, Y. Tay, A. Kumar, and H. Corporaal, “Transit: A visual analytical model for multithreaded machines,” in International Symposium on High-Performance Parallel and Distributed Computing (HPDC). ACM, 2015. find
  39. A. Li, S. L. Song, E. Brugel, A. Kumar, D. Chavarria-Miranda, and H. Corporaal, “X: A comprehensive analytic model for parallel machines,” in International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2016. find
  40. A. Li, W. Liu, M. R. Kristensen, B. Vinter, H. Wang, K. Hou, A. Marquez, and S. L. Song, “Exploring and analyzing the real impact of modern on-package memory on HPC scientific kernels,” in International Conference for High Performance Computing, Networking, Storage and Analysis (SC). ACM, 2017. find
  41. A. Li, G.-J. van den Braak, A. Kumar, and H. Corporaal, “Adaptive and transparent cache bypassing for GPUs,” in International Conference for High Performance Computing, Networking, Storage and Analysis (SC). ACM, 2015. find
  42. A. Li, G.-J. van den Braak, H. Corporaal, and A. Kumar, “Fine-grained synchronizations and dataflow programming on GPUs,” in International Conference on Supercomputing (ICS). ACM, 2015. find
  43. A. Li, S. L. Song, M. Wijtvliet, A. Kumar, and H. Corporaal, “SFU-driven transparent approximation acceleration on GPUs,” in International Conference on Supercomputing (ICS). ACM, 2016. find
  44. A. Li, S. L. Song, A. Kumar, E. Z. Zhang, D. Chavarrı́a-Miranda, and H. Corporaal, “Critical points based register-concurrency autotuning for GPUs,” in Conference on Design, Automation & Test in Europe, 2016. find
  45. W. Liu, A. Li, J. Hogg, I. S. Duff, and B. Vinter, “A synchronization-free algorithm for parallel sparse triangular solves,” in European Conference on Parallel Processing (EuroPar). Springer, 2016. find
  46. J. Li, Y. Ma, C. Yan, and R. Vuduc, “Optimizing sparse tensor times matrix on multi-core and many-core architectures,” in Proceedings of the Sixth Workshop on Irregular Applications: Architectures and Algorithms. IEEE Press, 2016. find
  47. A. Li, S. L. Song, W. Liu, X. Liu, A. Kumar, and H. Corporaal, “Locality-aware CTA clustering for modern GPUs,” ACM SIGOPS Operating Systems Review, 2017. find
  48. A. Li, W. Zhao, and S. L. Song, “Bvf: enabling significant on-chip power savings via bit-value-favor for throughput processors,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture. ACM, 2017. find
  49. A. Li, W. Liu, L. Wang, K. Barker, and S. L. Song, “Warp-consolidation: A novel execution model for gpus,” in International Conference on Supercomputing (ICS). ACM, 2018. find
  50. D. Shen, S. L. Song, A. Li, and X. Liu, “Cudaadvisor: Llvm-based runtime profiling for modern gpus,” in Proceedings of the 2018 International Symposium on Code Generation and Optimization. ACM, 2018. find
  51. L. Wang, J. Ye, Y. Zhao, W. Wu, A. Li, S. L. Song, Z. Xu, and T. Kraska, “Superneurons: Dynamic GPU memory management for training deep neural networks,” in ACM SIGPLAN Notices. ACM, 2018. find
  52. K. Spafford, J. S. Meredith, and J. S. Vetter, “Quantifying NUMA and contention effects in multi-GPU systems,” in GPGPU-4. ACM, 2011. find
  53. H. Wang, S. Potluri, D. Bureddy, C. Rosales, and D. K. Panda, “GPU-aware MPI on RDMA-enabled clusters: Design, implementation and evaluation,” IEEE Transactions on Parallel and Distributed Systems, 2014. find
  54. T. Ben-Nun, E. Levy, A. Barak, and E. Rubin, “Memory access patterns: the missing piece of the multi-GPU puzzle,” in International Conference for High Performance Computing, Networking, Storage and Analysis (SC). ACM, 2015. find
  55. J. Cabezas, L. Vilanova, I. Gelado, T. B. Jablin, N. Navarro, and W.-m. W. Hwu, “Automatic parallelization of kernels in shared-memory multi-GPU nodes,” in International Conference on Supercomputing (SC). ACM, 2015. find
  56. Y. Sun, S. Mukherjee, T. Baruah, S. Dong, J. Gutierrez, P. Mohan, and D. Kaeli, “Evaluating performance tradeoffs on the radeon open compute platform,” in International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2018. find
  57. H. Wang, S. Potluri, M. Luo, A. K. Singh, S. Sur, and D. K. Panda, “MVAPICH2-GPU: optimized GPU to GPU communication for InfiniBand clusters,” Computer Science-Research and Development, 2011. find
  58. T. Gysi, J. Bär, and T. Hoefler, “dCUDA: hardware supported overlap of computation and communication,” in International Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE, 2016. find
  59. B. Klenk, H. Fröening, H. Eberle, and L. Dennison, “Relaxations for high-performance message passing on massively parallel SIMT processors,” in International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2017. find
  60. A. A. Awan, C.-H. Chu, H. Subramoni, and D. K. Panda, “Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?” arXiv:1707.09414, 2017. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2019-evaluating-modern-gpu-interconnect-pcie/