📄 NVIDIA A100 Tensor Core GPU Architecture

Summary

This NVIDIA whitepaper (2020) describes the A100 Tensor Core GPU, built on the GA100 chip of the Ampere architecture and positioned as the eighth-generation data center GPU after Tesla V100. Modern clouds run very different workloads, including training, inference, HPC and analytics. NVIDIA wanted one accelerator that scales up for large jobs and can also be split across many small ones.

The main hardware changes are:

• third-generation Tensor Cores that support all common DL and HPC data types, including the new TF32 format;
• hardware acceleration for fine-grained 2:4 structured sparsity;
• 40 GB of HBM2 and a 40 MB L2 cache;
• third-generation NVLink and PCIe Gen 4 with SR-IOV;
• Multi-Instance GPU (MIG), which divides one card into isolated instances.

On the CUDA side, new features include asynchronous copy, asynchronous barriers, faster task graphs and L2 residency control.

On accuracy, an appendix reports that 2:4-sparse networks match their dense FP16 baselines across vision and NLP tasks. The A100 matters because it was the reference platform for large-scale model training at that time, for example in the DGX A100 system.

Key points

• The full GA100 chip has 128 SMs, of which the shipping A100 enables 108. The paper cites a 1410 MHz boost clock for its decode measurements.

• MIG can expose up to 7 isolated GPU Instances on a single A100, with fault isolation between them. Each GPU Instance can be further split into Compute Instances.

• 2:4 sparsity results (dense vs sparse FP16) from Table 11:

  Model	Metric	Dense	2:4 sparse
  ResNet-50	ImageNet Top-1	76.6	76.8
  MaskRCNN-ResNet-50	COCO bbox AP	37.9	37.9
  GNMT	WMT'14 En-De BLEU	24.6	24.9
  FairSeq Transformer	WMT'14 En-De BLEU	28.2	28.5
  BERT Large	SQuAD v1.1 F1	91.1	91.5
  Transformer XL	enwik8 BPC	1.06	1.06

• Mixed-precision Tensor Cores are applied to HPC through the TCAIRS solver. It reaches FP64-level accuracy faster than a baseline FP64 direct solver.

• The chip also includes fixed-function NVJPG and NVDEC decoders and an Optical Flow Accelerator to feed DL pipelines. The DGX A100 system combines eight A100 GPUs.

Interesting ideas

• I1. Prune, then fine-tune into a regular pattern. The 2:4 scheme keeps 2 of every 4 weights, so the sparsity is regular enough for hardware to exploit, unlike arbitrary pruning. The paper prunes trained dense networks and fine-tunes them to recover accuracy. This recipe could be tried for compressing our NLP models before deployment.
• I2. Cheap precision first, refine afterwards. TCAIRS does the heavy work in low precision and then refines iteratively to full FP64 accuracy. The same principle could apply wherever a fast approximate step can be corrected by a cheap exact one.
• I3. Hard partitioning instead of time-sharing. MIG gives each tenant dedicated, isolated resources, which yields predictable latency and fault containment. This is a useful design model for sharing one expensive GPU among several small inference or annotation services.
• I4. Hide data movement behind compute. Asynchronous copy, asynchronous barriers and L2 residency control let programs prefetch data and keep it close to the cores while computation continues. The general lesson for data pipelines is to overlap loading with processing.
• I5. Report compression on varied tasks with native metrics. Table 11 checks the sparsity trade-off across classification, detection, translation and language modelling, each with its own metric (Top-1, bbox AP, BLEU, BPC, F1). This makes the "no accuracy loss" claim far more convincing than a single benchmark would.