PyTorch: An Imperative Style, High-Performance Deep Learning Library

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

3466 papers of our library cite it (of the 140673 whose bibliographies we read); 59 of them have a page here, 3407 not yet.

• gpt-oss-120b & gpt-oss-20b Model Card
• Refusal in Language Models Is Mediated by a Single Direction
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns
• Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
• On the Opportunities and Risks of Foundation Models
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
• RobBERT: a Dutch RoBERTa-based Language Model
• Few-NERD: A Few-shot Named Entity Recognition Dataset
• Word-Level Coreference Resolution
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
• Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary
• HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
• Efficient Attentions for Long Document Summarization
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
• Editing Models with Task Arithmetic
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task
• MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Large language models are zero-shot reasoners
• PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Symbolic Chain-of-Thought Distillation: Small Models Can Also ``Think'' Step-by-Step
• DataComp-LM: In search of the next generation of training sets for language models
• Noisy Channel Language Model Prompting for Few-Shot Text Classification
• Learning to Compress Prompts with Gist Tokens
• Generative Representational Instruction Tuning
• Progress measures for grokking via mechanistic interpretability
• A Comprehensive Overview of Large Language Models
• LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution
• KernelBench: Can LLMs Write Efficient GPU Kernels?
• True Few-Shot Learning with Language Models
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
• Learning Transferable Visual Models From Natural Language Supervision
• Robust Speech Recognition via Large-Scale Weak Supervision
• "You might think about slightly revising the title": Identifying Hedges in Peer-tutoring Interactions
• Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants
• Improving Passage Retrieval with Zero-Shot Question Generation
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• HybridFlow: A Flexible and Efficient RLHF Framework
• ViperGPT: Visual Inference via Python Execution for Reasoning
• Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
• Training data-efficient image transformers & distillation through attention
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Are All Languages Created Equal in Multilingual BERT?
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• FUDGE: Controlled Text Generation With Future Discriminators
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• A Survey of Large Language Models
• SGLang: Efficient Execution of Structured Language Model Programs
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.01703 · arXiv 1912.01703

@misc{paszke2019pytorch,
  title = {{PyTorch: An Imperative Style, High-Performance Deep Learning Library}},
  author = {Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Köpf, Andreas and Yang, Edward and DeVito, Zach and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith},
  year = {2019},
  doi = {10.48550/arxiv.1912.01703},
  eprint = {1912.01703},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.01703}
}

References

44 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yangqing "Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor" Darrell. "caffe: Convolutional architecture for fast feature embedding". "arXiv preprint arXiv:1408.5093", "2014". find our card
2. Frank Seide and Amit Agarwal. Cntk: Microsoft’s open-source deep-learning toolkit. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 2135–2135, New York, NY, USA, 2016. ACM. find
3. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenber… find
4. Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016. find our card
5. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), 2015. find
6. Ronan Collobert, Samy Bengio, and Johnny Mariéthoz. Torch: a modular machine learning software library. Technical report, Idiap, 2002. find
7. G. Neubig, C. Dyer, Y. Goldberg, A. Matthews, W. Ammar, A. Anastasopoulos, M. Ballesteros, D. Chiang, D. Clothiaux, T. Cohn, K. Duh, M. Faruqui, C. Gan, D. Garrette, Y. Ji, L. Kong, A. Kuncoro, G. Kumar, C. Malaviya, P. Michel, Y. Oda, M. Richardson, N. Saphra, S. Swayamdipta, and P. Yin. DyNet: The… find our card
8. Philip S. Abrams. An APL Machine. PhD thesis, Stanford University, 1970. find
9. The MathWorks, Inc., Natick, Massachusetts, United States. MATLAB and Statistics Toolbox. find
10. R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. find
11. Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65–98, 2017. find
12. Travis Oliphant. NumPy: http://www.numpy.org/. A guide to NumPy. ↗
13. Gaël Guennebaud, Benoît Jacob, et al. Eigen v3. http://eigen.tuxfamily.org, 2010. ↗
14. Y LeCun and L Bottou. Lush reference manual. http://lush.sourceforge.net, 2002. ↗
15. Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: A survey. J. Mach. Learn. Res., 18(1):5595–5637, January 2017. find
16. Dougal Maclaurin. Modeling, Inference and Optimization with Composable Differentiable Procedures. PhD thesis, Harvard University, April 2016. find
17. Matthew Johnson et. al. Jax. https://github.com/google/jax, 2018. ↗
18. Mike Innes et. al. Flux.jl. https://github.com/FluxML/Flux.jl, 2018. ↗
19. Eric Jones, Travis Oliphant, Pearu Peterson, et al. SciPy: Open source scientific tools for Python, 2001–. http://www.scipy.org/. ↗
20. Wes McKinney. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, 51-56, 2010. find
21. Pierre Sermanet, Koray Kavukcuoglu, and Yann LeCun. Eblearn: Open-source energy-based learning in c++. In 2009 21st IEEE International Conference on Tools with Artificial Intelligence, pages 693–697. IEEE, 2009. USA: Trelgol Publishing, 2006. Technical report, code available at find
22. Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan D. Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. CoRR, abs/1410.0759, 2014. find
23. Andrew Lavin. maxdnn: An efficient convolution kernel for deep learning with maxwell gpus, January 2015. find
24. Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4013–4021, 2016. find
25. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In NIPS 2011, 2011. find
26. Richard Gabriel. The rise of worse is better. http://dreamsongs.com/RiseOfWorseIsBetter.html. ↗
27. Yann LeCun and Corinna Cortes. http://yann.lecun.com/exdb/mnist/. ↗
28. Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy P. Lillicrap,… — [arXiv:1708.04782](https://arxiv.org/abs/1708.04782) find
29. DMLC. Dlpack: Open in memory tensor structure. https://github.com/dmlc/dlpack. ↗
30. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS Workshop, 2017. find
31. Dan Piponi. Automatic differentiation, C++ templates, and photogrammetry. J. Graphics, GPU, & Game Tools, 9(4):41–55, 2004. find
32. Holger Leuck and Hans-Hellmut Nagel. Automatic differentiation facilitates of-integration into steering-angle-based road vehicle tracking. In 1999 Conference on Computer Vision and Pattern Recognition (CVPR ’99), 23-25 June 1999, Ft. Collins, CO, USA, pages 2360–2365, 1999. find
33. The Python team. The cpython https://wiki.python.org/moin/GlobalInterpreterLock. ↗
34. Giovanni Petrantoni and Jörg Wollenschläger. xyz/nimtorch. find
35. Austin Huang, Junji Hashimoto, https://github.com/hasktorch/hasktorch. ↗
36. G. Synnaeve, Z. Lin, J. Gehring, D. Gant, V. Mella, V. Khalidov, N. Carion, and N. Usunier. Forward modeling for partial observation strategy games - a starcraft defogger. In Advances in Neural Information Processing Systems, pages 10761–10771, 2018. find
37. The PyTorch team. Torch Script. https://pytorch.org/docs/stable/jit.html. ↗
38. Justin Luitjens. Cuda streams. GPU technology conference, 2014. find
39. Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe, and Paul R. Wilson. Hoard: A scalable memory allocator for multithreaded applications. In Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS IX, pages 117–128, New… find
40. J. Evans. A scalable concurrent malloc(3) implementation for freebsd. In In BSDCan — The Technical BSD Conference, May 2006. find
41. S. Ghemawat and P. Menage. Tcmalloc: Thread-caching malloc. MNIST handwritten digit database. global interpreter lock. Nimtorch. https://github.com/fragcolor-and Sam Stites. Hasktorch. ↗
42. Benjamin Recht, Christopher Ré, Stephen J. Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12… find
43. Matthew Hertz and Emery D. Berger. Quantifying the performance of garbage collection vs. explicit memory management. In Proceedings of the 20th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, OOPSLA ’05, pages 313–326, New York, NY, USA, 2005. ACM. find
44. The PyTorch team. Pytorch Autograd Profiler. https://pytorch.org/docs/1.0.1/autograd.html\#profiler. ↗