Elamur Research
☰
home / papers / 2018 / TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1802.04799

Summary

Deploying deep learning models to anything other than mainstream server GPUs usually requires a lot of manual work. Existing frameworks hand individual operators to vendor-tuned libraries that are hard to port. They also force a trade-off: either skip graph rewrites that create new operators, or run those operators unoptimized. TVM addresses this with one compiler that optimizes jointly at two levels. At the graph level, it applies operator fusion, constant folding, static memory planning and layout changes. At the operator level, a declarative tensor expression language builds on Halide's split between computation and scheduling. It adds new schedule primitives for cross-thread memory reuse, tensorization with hardware intrinsics, and latency hiding on accelerators. Rather than relying on a hand-written cost model, a learned model guides the search over candidate implementations and improves as hardware measurements accumulate. The evaluation spanned four targets: a GPU for servers, a GPU and a CPU intended for embedded use, and an accelerator built on an FPGA. TVM matched or exceeded hand-tuned libraries, which makes efficient compilation for new hardware far less labor-intensive.

Key points

  • Reported end-to-end speedups range from 1.2× to 3.8× over frameworks that rely on hand-optimized operator libraries.
  • Models can be imported from TensorFlow, MXNet, PyTorch, Keras and CNTK, as well as the CoreML and ONNX formats. Code is generated through LLVM IR, CUDA, Metal or OpenCL, or for an accelerator back-end.
  • The fusion rules sort operators into four classes: injective (e.g., add), reduction (e.g., sum), complex-out-fusable (e.g., conv2d) and opaque (e.g., sort).
  • Accelerator support was demonstrated on VDLA, a generic FPGA-based design that models TPU-like hardware. The compiler explicitly manages pipeline dependencies to hide memory latency.
  • TVM is open source and, per the authors, used in production at several major companies. The paper appeared at OSDI 2018 (arXiv:1802.04799).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

415 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 412 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1802.04799 · arXiv 1802.04799

@misc{chen2018tvm,
  title = {{TVM: An Automated End-to-End Optimizing Compiler for Deep Learning}},
  author = {Chen, Tianqi and Moreau, Thierry and Jiang, Ziheng and Zheng, Lianmin and Yan, Eddie and Cowan, Meghan and Shen, Haichen and Wang, Leyuan and Hu, Yuwei and Ceze, Luis and Guestrin, Carlos and Krishnamurthy, Arvind},
  year = {2018},
  doi = {10.48550/arxiv.1802.04799},
  eprint = {1802.04799},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.04799}
}

References

48 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. NVIDIA Tesla V100 GPU Architecture: The World’s Most Advanced Data Center GPU, 2017. find
  2. A BADI , M., A GARWAL , A., B ARHAM , P., B REVDO , E., C HEN , Z., C ITRO , C., C ORRADO , G. S., D AVIS , A., D EAN , J., D EVIN , M., G HEMAWAT , S., G OODFELLOW , I., H ARP , A., I RVING , G., I SARD , M., J IA , Y., J OZEFOWICZ , R., K AISER , L., K UDLUR , M., L EVENBERG , J., M AN É , D., M… find
  3. A BADI , M., B ARHAM , P., C HEN , J., C HEN , Z., D AVIS , A., D EAN , J., D EVIN , M., G HEMAWAT , S., I RVING , G., I SARD , M., K UDLUR , M., L EVENBERG , J., M ONGA , R., M OORE , S., M URRAY , D. G., S TEINER , B., T UCKER , P., V ASUDEVAN , V., W ARDEN , P., W ICKE , M., Y U , Y., AND Z HENG… find
  4. A GARWAL , A., A KCHURIN , E., B ASOGLU , C., C HEN , G., C YPHERS , S., D ROPPO , J., E VERSOLE , A., G UENTER , B., H ILLEBRAND , M., H OENS , R., H UANG , X., H UANG , Z., I VANOV , V., K AMENEV , A., K RANEN , P., K UCHAIEV , O., M ANOUSEK , W., M AY , A., M ITRA , B., N ANO , O., N AVARRO , G.,… find
  5. A NSEL , J., K AMIL , S., V EERAMACHANENI , K., R AGAN - K ELLEY , J., B OSBOOM , J., O’R EILLY , U.-M., AND A MARAS - INGHE , S. Opentuner: An extensible framework for program autotuning. In International Conference on Parallel Architectures and Compilation Techniques (Edmonton, Canada, August 2014… find
  6. B AGHDADI , R., B EAUGNON , U., C OHEN , A., G ROSSER , T., K RUSE , M., R EDDY , C., V ERDOOLAEGE , S., B ETTS , A., D ONALDSON , A. F., K ETEMA , J., A BSAR , J., H AASTREGT , S. V ., K RAVETS , A., L OKHMOTOV , A., D AVID , R., AND H A - JIYEV , E. Pencil: A platform-neutral compute intermediate… find
  7. B ASTIEN , F., L AMBLIN , P., P ASCANU , R., B ERGSTRA , J., G OODFELLOW , I. J., B ERGERON , A., B OUCHARD , N., AND B ENGIO , Y. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012. find
  8. C HEN , T., AND G UESTRIN , C. Xgboost: A scalable tree boosting system. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA, 2016), KDD ’16, ACM, pp. 785–794. find
  9. C HEN , T., L I , M., L I , Y., L IN , M., W ANG , N., W ANG , M., X IAO , T., X U , B., Z HANG , C., , AND Z HANG , Z. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. In Neural Information Processing Systems, Workshop on Machine Learning Systems (Lear… find our card
  10. C HEN , T.-F., AND B AER , J.-L. Effective hardware-based data prefetching for high-performance processors. IEEE Transactions on Computers 44, 5 (May 1995), 609–623. find
  11. C HEN , Y., L UO , T., L IU , S., Z HANG , S., H E , L., W ANG , J., L I , L., C HEN , T., X U , Z., S UN , N., AND T EMAM , O. Dadiannao: A machine-learning supercomputer. In Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture (Washington, DC, USA, 2014), MICRO-47,… find
  12. C HEN , Y.-H., E MER , J., AND S ZE , V. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. In Proceedings of the 43rd International Symposium on Computer Architecture (Piscataway, NJ, USA, 2016), ISCA ’16, IEEE Press, pp. 367–379. find
  13. C OURBARIAUX , M., B ENGIO , Y., AND D AVID , J. Binaryconnect: Training deep neural networks with binary weights during propagations. CoRR abs/1511.00363 (2015). find
  14. E GGERS , S. J., E MER , J. S., L EVY , H. M., L O , J. L., S TAMM , R. L., AND T ULLSEN , D. M. Simultaneous multithreading: a platform for next-generation processors. IEEE Micro 17, 5 (Sept 1997), 12–19. find
  15. F RIGO , M., AND J OHNSON , S. G. Fftw: an adaptive software architecture for the fft. In Acoustics, Speech and Signal Processing, 1998. Proceedings of the 1998 IEEE International Conference on (May 1998), vol. 3, pp. 1381–1384 vol.3. find
  16. H E , K., Z HANG , X., R EN , S., AND S UN , J. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027 (2016). find
  17. H EGARTY , J., B RUNHAVER , J., D E V ITO , Z., R AGAN -K ELLEY , J., C OHEN , N., B ELL , S., V ASILYEV , A., H OROWITZ , M., AND H ANRAHAN , P. Darkroom: Compiling high-level image processing code into hardware pipelines. ACM Trans. Graph. 33, 4 (July 2014), 144:1–144:11. find
  18. H ENRIKSEN , T., S ERUP , N. G. W., E LSMAN , M., H ENGLEIN , F., AND O ANCEA , C. E. Futhark: Purely functional gpu-programming with nested parallelism and in-place array updates. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (New York, NY, USA,… find
  19. H OWARD , A. G., Z HU , M., C HEN , B., K ALENICHENKO , D., W ANG , W., W EYAND , T., A NDREETTO , M., AND A DAM , H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017). find our card
  20. J OUPPI , N. P. Improving direct-mapped cache performance by the addition of a small fully-associative cache and prefetch buffers. In [1990] Proceedings. The 17th Annual International Symposium on Computer Architecture (May 1990), pp. 364–373. find
  21. J OUPPI , N. P., Y OUNG , C., P ATIL , N., P ATTERSON , D., A GRAWAL , G., B AJWA , R., B ATES , S., B HATIA , S., B ODEN , N., B ORCHERS , A., B OYLE , R., C ANTIN , P.- L ., C HAO , C., C LARK , C., C ORIELL , J., D ALEY , M., D AU , M., D EAN , J., G ELB , B., G HAEMMAGHAMI , T. V., G OTTIPATI ,… find
  22. K IRKPATRICK , S., G ELATT , C. D., AND V ECCHI , M. P. Optimization by simulated annealing. Science 220, 4598 (1983), 671–680. find
  23. K JOLSTAD , F., K AMIL , S., C HOU , S., L UGATO , D., AND A MARASINGHE , S. The tensor algebra compiler. Proc. ACM Program. Lang. 1, OOPSLA (Oct. 2017), 77:1–77:29. find
  24. K L ÖCKNER , A. Loo.py: transformation-based code generation for GPUs and CPUs. In Proceedings of ARRAY ‘14: ACM SIG-PLAN Workshop on Libraries, Languages, and Compilers for Array Programming (Edinburgh, Scotland., 2014), Association for Computing Machinery. find
  25. L AVIN , A., AND G RAY , S. Fast algorithms for convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 (2016), pp. 4013–4021. find
  26. L I , L., J AMIESON , K. G., D E S ALVO , G., R OSTAMIZADEH , A., AND T ALWALKAR , A. Efficient hyperparameter optimization and infinitely many armed bandits. CoRR abs/1603.06560 (2016). — [arXiv:1603.06560](https://arxiv.org/abs/1603.06560) find
  27. L IU , D., C HEN , T., L IU , S., Z HOU , J., Z HOU , S., T EMAN , O., F ENG , X., Z HOU , X., AND C HEN , Y. Pudiannao: A polyvalent machine learning accelerator. In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (New Y… find
  28. M NIH , V., K AVUKCUOGLU , K., S ILVER , D., R USU , A. A., V ENESS , J., B ELLEMARE , M. G., G RAVES , A., R IEDMILLER , M., F IDJELAND , A. K., O STROVSKI , G., ET AL . Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529. find
  29. M ULLAPUDI , R. T., A DAMS , A., S HARLET , D., R AGAN - K ELLEY , J., AND F ATAHALIAN , K. Automatically scheduling halide image processing pipelines. ACM Trans. Graph. 35, 4 (July 2016), 83:1–83:11. find
  30. P ALKAR , S., T HOMAS , J. J., N ARAYANAN , D., S HANBHAG , A., P ALAMUTTAM , R., P IRK , H., S CHWARZKOPF , M., A MA - RASINGHE , S. P., M ADDEN , S., AND Z AHARIA , M. Weld: Rethinking the interface between data-intensive applications. CoRR abs/1709.06416 (2017). — [arXiv:1709.06416](https://arxiv.org/abs/1709.06416) find
  31. R ADFORD , A., M ETZ , L., AND C HINTALA , S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015). find our card
  32. R AGAN -K ELLEY , J., B ARNES , C., A DAMS , A., P ARIS , S., D U - RAND , F., AND A MARASINGHE , S. Halide: A language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design… find
  33. R ASTEGARI , M., O RDONEZ , V., R EDMON , J., AND F ARHADI , A. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision (2016), Springer, pp. 525–542. find
  34. S HARMA , H., P ARK , J., M AHAJAN , D., A MARO , E., K IM , J. K., S HAO , C., M ISHRA , A., AND E SMAEILZADEH , H. From high-level deep neural models to fpgas. In Microarchitecture (MI-CRO), 2016 49th Annual IEEE/ACM International Symposium on (2016), IEEE, pp. 1–12. find
  35. S MITH , J. E. Decoupled access/execute computer architectures. In Proceedings of the 9th Annual Symposium on Computer Architecture (Los Alamitos, CA, USA, 1982), ISCA ’82, IEEE Computer Society Press, pp. 112–119. find
  36. S TEUWER , M., R EMMELG , T., AND D UBACH , C. Lift: A functional data-parallel ir for high-performance gpu code generation. In Proceedings of the 2017 International Symposium on Code Generation and Optimization (Piscataway, NJ, USA, 2017), CGO ’17, IEEE Press, pp. 74–85. find
  37. S UJEETH , A. K., L EE , H., B ROWN , K. J., C HAFI , H., W U , M., A TREYA , A. R., O LUKOTUN , K., R OMPF , T., AND O DERSKY , M. Optiml: An implicitly parallel domain-specific language for machine learning. In Proceedings of the 28th International Conference on International Conference on Machine… find
  38. T AI , K. S., S OCHER , R., AND M ANNING , C. D. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075 (2015). find
  39. T ULLOCH , A., AND J IA , Y. High performance ultra-low-precision convolutions on mobile devices. arXiv preprint arXiv:1712.02427 (2017). — [arXiv:1712.02427](https://arxiv.org/abs/1712.02427) find
  40. U MUROGLU , Y., F RASER , N. J., G AMBARDELLA , G., B LOTT , M., L EONG , P. H. W., J AHRE , M., AND V ISSERS , K. A. FINN: A framework for fast, scalable binarized neural network inference. CoRR abs/1612.07119 (2016). — [arXiv:1612.07119](https://arxiv.org/abs/1612.07119) find
  41. V ASILACHE , N. personal communication. find
  42. V ASILACHE , N., Z INENKO , O., T HEODORIDIS , T., G OYAL , P., D E V ITO , Z., M OSES , W. S., V ERDOOLAEGE , S., A DAMS , A., AND C OHEN , A. Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions. CoRR abs/1802.04730 (2018). find
  43. V ERDOOLAEGE , S., C ARLOS J UEGA , J., C OHEN , A., I GNA - CIO G ÓMEZ , J., T ENLLADO , C., AND C ATTHOOR , F. Polyhedral parallel code generation for cuda. ACM Trans. Archit. Code Optim. 9, 4 (Jan. 2013), 54:1–54:23. find
  44. V OLKOV , V. Understanding Latency Hiding on GPUs. PhD thesis, University of California at Berkeley, 2016. find
  45. W EI , R., A DVE , V., AND S CHWARTZ , L. Dlvm: A modern compiler infrastructure for deep learning systems. CoRR abs/1711.03016 (2017). find
  46. W HALEY , R. C., AND D ONGARRA , J. J. Automatically tuned linear algebra software. In Proceedings of the 1998 ACM/IEEE Conference on Supercomputing (Washington, DC, USA, 1998), SC ’98, IEEE Computer Society, pp. 1–27. find
  47. W ILLIAMS , S., W ATERMAN , A., AND P ATTERSON , D. Roofline: An insightful visual performance model for multicore architectures. Commun. ACM 52, 4 (Apr. 2009), 65–76. find
  48. Z AREMBA , W., S UTSKEVER , I., AND V INYALS , O. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014). find our card