In-Datacenter Performance Analysis of a Tensor Processing Unit
Paper · 2017 · Norman P. Jouppi; Cliff Young; Nishant Patil; David Patterson; Gaurav Agrawal; Raminder Bajwa; Sarah Bates; Suresh Bhatia; Nan Boden; Al Borchers; Rick Boyle; Pierre-luc Cantin; Clifford Chao; Chris Clark; Jeremy Coriell; Mike Daley; Matt Dau; Jeffrey Dean; Ben Gelb; Tara Vazir Ghaemmaghami; Rajendra Gottipati; William Gulland; Robert Hagmann; C. Richard Ho; Doug Hogberg; John Hu; Robert Hundt; Dan Hurt; Julian Ibarz; Aaron Jaffey; Alek Jaworski; Alexander Kaplan; Harshit Khaitan; Andy Koch; Naveen Kumar; Steve Lacy; James Laudon; James Law; Diemthu Le; Chris Leary; Zhuyuan Liu; Kyle Lucke; Alan Lundin; Gordon MacKean; Adriana Maggiore; Maire Mahony; Kieran Miller; Rahul Nagarajan; Ravi Narayanaswami; Ray Ni; Kathy Nix; Thomas Norrie; Mark Omernick; Narayana Penukonda; Andy Phelps; Jonathan Ross; Matt Ross; Amir Salek; Emad Samadiani; Chris Severn; Gregory Sizikov; Matthew Snelham; Jed Souter; Dan Steinberg; Andy Swing; Mercedes Tan; Gregory Thorson; Bo Tian; Horia Toma; Erick Tuttle; Vijay Vasudevan; Richard Walter; Walter Wang; Eric Wilcox; and Doe Hyun Yoon
Summary
By 2013, Google projected that neural network inference could force a costly expansion of its datacenters. One estimate assumed users ran speech-recognition voice search for a few minutes per day and found that capacity would need to double. Rather than buy more CPUs, the team designed and deployed a custom inference ASIC, the Tensor Processing Unit (TPU), in about 15 months. It went into service in 2015 as a PCIe coprocessor.
The chip centres on a 256x256 grid of 8-bit multiply-accumulate units, backed by 28 MiB of on-chip memory that software manages directly. It uses a simple, deterministic execution model and drops general-purpose hardware such as caches and out-of-order logic.
The authors compare it with a Haswell CPU and an Nvidia K80 GPU in the same datacenters. The test set is six production TensorFlow models (MLPs, LSTMs, CNNs) that together cover 95% of inference demand, run under strict tail-latency limits. The TPU is about 15–30X faster and 30–80X more energy-efficient. With GPU-class memory, its efficiency advantage could approach 70X over the GPU and 200X over the CPU.
Key points
- Compute and memory: the Matrix Multiply Unit holds 65,536 8-bit MACs with a peak of 92 TOPS.
- On-chip storage is 28 MiB in total: a 24 MiB Unified Buffer plus 4 MiB of 32-bit Accumulators.
- Weights come from an 8 GiB off-chip DRAM (Weight Memory).
- Benchmarks: six production models (MLP0/1, LSTM0/1, CNN0/1), each only 100–1500 lines of TensorFlow and using 5M–100M weights.
- The set includes RankBrain, part of GNM Translate, Inception and AlphaGo.
- CNNs account for only about 5% of the deployed workload.
- Die area: data buffers take 37% of the die, compute 30%, I/O 10% and control just 2%.
- Instruction set: the host sends CISC-style instructions over PCIe Gen3 x16, about a dozen in total.
- Each takes roughly 10–20 cycles, and the set includes a repeat field.
- Bottlenecks: four of the six applications are limited by memory bandwidth on the TPU.
- CPU vs. GPU: under latency limits, the K80 is only slightly faster than Haswell for inference.
Interesting ideas
- I1. Low utilization can still pay off. The authors call this the "Cornucopia Corollary": running a huge, cheap resource at a small fraction of its capacity can still beat expensive general-purpose hardware. This is a useful way to argue for wide, simple designs over heavily optimized ones in any cost–performance trade-off.
- I2. Judge systems by tail latency, not averages. Hardware is compared at its 99th-percentile response-time limits rather than at average throughput. That shift is what reveals the GPU's weakness for user-facing inference, and the same logic applies to evaluating any interactive service, such as NLP model serving.
- I3. Size hardware from a performance model. The accumulator count was taken from the roofline "knee" (operations per byte needed to reach peak), rounded up and then doubled to allow double buffering. Deriving resource sizes from such a model is reusable when provisioning buffers, batch sizes or caches elsewhere.
- I4. Measure the real workload before optimizing. Profiling the actual datacenter mix showed that CNNs, the focus of most accelerator research, were only about 5% of demand. Profiling real usage before choosing optimization targets applies equally to data pipelines and model serving.
- I5. Prefer evaluation metrics that do not depend on the model. Inferences per second depends more on the network than on the hardware, so it is a poor single metric. When comparing systems, choose metrics that do not mostly reflect the particular model or dataset being run.
Institutions
United StatesFields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works
Cite
Cite as Norman P. Jouppi; Cliff Young; Nishant Patil; David Patterson; Gaurav Agrawal; Raminder Bajwa; Sarah Bates; Suresh Bhatia; Nan Boden; Al Borchers; Rick Boyle; Pierre-luc Cantin; Clifford Chao; Chris Clark; Jeremy Coriell; Mike Daley; Matt Dau; Jeffrey Dean; Ben Gelb; Tara Vazir Ghaemmaghami; Rajendra Gottipati; William Gulland; Robert Hagmann; C. Richard Ho; Doug Hogberg; John Hu; Robert Hundt; Dan Hurt; Julian Ibarz; Aaron Jaffey; Alek Jaworski; Alexander Kaplan; Harshit Khaitan; Andy Koch; Naveen Kumar; Steve Lacy; James Laudon; James Law; Diemthu Le; Chris Leary; Zhuyuan Liu; Kyle Lucke; Alan Lundin; Gordon MacKean; Adriana Maggiore; Maire Mahony; Kieran Miller; Rahul Nagarajan; Ravi Narayanaswami; Ray Ni; Kathy Nix; Thomas Norrie; Mark Omernick; Narayana Penukonda; Andy Phelps; Jonathan Ross; Matt Ross; Amir Salek; Emad Samadiani; Chris Severn; Gregory Sizikov; Matthew Snelham; Jed Souter; Dan Steinberg; Andy Swing; Mercedes Tan; Gregory Thorson; Bo Tian; Horia Toma; Erick Tuttle; Vijay Vasudevan; Richard Walter; Walter Wang; Eric Wilcox; and Doe Hyun Yoon. 2017. In-Datacenter Performance Analysis of a Tensor Processing Unit. arXiv:1704.04760.
@misc{wilcox2017datacenter,
title = {{In-Datacenter Performance Analysis of a Tensor Processing Unit}},
author = {Norman P. Jouppi; Cliff Young; Nishant Patil; David Patterson; Gaurav Agrawal; Raminder Bajwa; Sarah Bates; Suresh Bhatia; Nan Boden; {Al Borchers; Rick Boyle; Pierre-luc Cantin; Clifford Chao; Chris Clark; Jeremy Coriell; Mike Daley; Matt Dau; Jeffrey Dean; Ben Gelb; Tara Vazir Ghaemmaghami; Rajendra Gottipati; William Gulland; Robert Hagmann; C. Richard Ho; Doug Hogberg; John Hu; Robert Hundt; Dan Hurt; Julian Ibarz; Aaron Jaffey; Alek Jaworski; Alexander Kaplan; Harshit Khaitan; Andy Koch; Naveen Kumar; Steve Lacy; James Laudon; James Law; Diemthu Le; Chris Leary; Zhuyuan Liu; Kyle Lucke; Alan Lundin; Gordon MacKean; Adriana Maggiore; Maire Mahony; Kieran Miller; Rahul Nagarajan; Ravi Narayanaswami; Ray Ni; Kathy Nix; Thomas Norrie; Mark Omernick; Narayana Penukonda; Andy Phelps; Jonathan Ross; Matt Ross; Amir Salek; Emad Samadiani; Chris Severn; Gregory Sizikov; Matthew Snelham; Jed Souter; Dan Steinberg; Andy Swing; Mercedes Tan; Gregory Thorson; Bo Tian; Horia Toma; Erick Tuttle; Vijay Vasudevan; Richard Walter; Walter Wang; Eric Wilcox;} and Doe Hyun Yoon},
year = {2017},
doi = {10.48550/arxiv.1704.04760},
eprint = {1704.04760},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1704.04760}
}References
- Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M. and Ghemawat, S., 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467. Albericio, J., Judd, P., Hetherington, T., Aamodt…
- LiKamWa, R., Hou, Y., Gao, J., Polansky, M. and Zhong, L., 2016. RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision. Proceedings of the 43rd International Symposium on Computer Architecture. Liu, D., Chen, T., Liu, S., Zhou, J., Zhou, S., Teman, O., Feng, X., Zhou, X. an… find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/jouppi-2017-datacenter-performance-analysis-tensor-processing/