📄 NVIDIA Tensor Core Programmability, Performance & Precision

Summary

The paper asks whether traditional HPC codes, not just deep learning, can benefit from the Tensor Cores in NVIDIA's Volta GPUs. These units perform a 4×4 matrix-multiply-and-accumulate each clock cycle. They take half-precision inputs and accumulate in single precision. The authors study three questions on a Tesla V100:

• Programmability: they compare the three available interfaces, from lowest to highest level. These are the CUDA 9 WMMA API, the templated CUTLASS library built on WMMA, and cuBLAS GEMM.
• Performance: they benchmark large GEMMs and batched small GEMMs, and compare the results with ordinary CUDA cores.
• Precision: they measure how much accuracy is lost when inputs are rounded to half precision, across different matrix sizes. They also propose a way to reduce that loss by doing extra computation.

The best measured throughput is about 83 Tflops/s in mixed precision. That is roughly seven times single-precision speed and three times half-precision speed. These results matter because upcoming supercomputers such as Summit and Sierra rely on V100 GPUs. HPC developers therefore need concrete guidance on how to program these units and what accuracy to expect.

Key points

• Hardware: a Tesla V100 has 80 SMs and 640 Tensor Cores. Each Tensor Core performs 64 FMA operations per cycle.
• Theoretical peaks: 125 Tflops/s for Tensor Cores, compared with 31.4 (FP16), 15.7 (FP32) and 7.8 (FP64) Tflops/s on the regular cores.
• Scale: Summit, with six V100s per node across about 4,600 nodes, will contain nearly 18M Tensor Cores.
• WMMA: the API works on 16×16 tiles (non-square shapes from CUDA 9.1) per warp of 32 threads. It is a preview feature with no backward-compatibility guarantee.
• Measured results: large GEMMs reach about 83 Tflops/s. A batched GEMM written with WMMA reaches about 4 Tflops/s.

Interesting ideas

• I1. Compare abstraction levels on one operation. The authors benchmark the same GEMM through a low-level API, a template library and a vendor BLAS. This separates the cost of expressiveness from raw speed, and the same design works for evaluating any new accelerator or library stack.
• I2. Overdecompose the problem. WMMA exposes only tiles larger than the 4×4 hardware unit, so many more threads run than there are compute units. The extra threads help hide memory and instruction latency, a principle that applies to other parallel kernels too.
• I3. Report accuracy as a function of problem size. Measuring error against matrix size, rather than at one size, shows how low-precision error grows. This is a useful evaluation habit whenever reduced-precision arithmetic or quantization is considered.
• I4. Trade compute for accuracy. When the hardware's native precision is too low, spending extra fast low-precision operations can recover much of the lost accuracy. This trade-off is worth weighing for any quantized pipeline.
• I5. Benchmark batched small problems separately. Batched small GEMMs behave very differently from one large GEMM, so they deserve their own tests. This matters for workloads made of many small independent operations.