🧑‍🔬 Deepak Narayanan

🏛️ Affiliations

🏢NvidiaSanta Clara · United States2023–2026 · 5 paperscurrent🏢NvidiaReading · United Kingdom2023–2026 · 4 paperscurrent🎓Stanford UniversityStanford · United States2017–2022 · 13 papers🏢MicrosoftRedmond · United States2020–2021 · 3 papers

From the bylines of their papers (OpenAlex, 2026-10-14), ORCID and Wikidata; institutions on papers are matched by machine. 3 more institutions on single bylines are not listed.

🧭 Fields, topics and concepts

Of its works in our library, with the number of works on each.

📅 Works by year

2020s

2010s

📄 Papers

The most cited first (OpenAlex and our library).

1. PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018 · 162 citations)
   Data-parallel DNN training gives every GPU a full model copy and synchronizes weights after each step. As models grow and GPUs get faster, network traffic increasingly dominates runtime. For VGG16, it can take up to 85% of training time. Classic model parallelism avoids exchanging all weights but leaves most workers…
2. Memory-Efficient Pipeline-Parallel DNN Training (2020 · 153 citations)
   Very large language models often cannot fit their weights and activations on one accelerator, so training must be spread across devices.
3. Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads (2020 · 141 citations)
   Deep learning training clusters now mix accelerators of different generations, and models benefit unequally from newer hardware.
4. An Empirical Study of Mamba-based Language Models (2024 · 67 citations)
   Selective state-space models (SSMs) such as Mamba could make inference cheaper than in Transformers. They avoid attention's quadratic cost and the ever-growing key-value cache. However, earlier comparisons with identical training data covered only small models. This NVIDIA-led report addresses that gap. It trains…
5. Nemotron-4 15B Technical Report (2024 · 30 citations)
   NVIDIA set out to build the strongest general-purpose LLM that fits on a single A100 or H100 GPU. Following the Chinchilla-inspired idea of spending compute on more data rather than more parameters, the authors trained a 15B-parameter decoder-only Transformer on 8 trillion tokens. The mix was about 70% English, 15%…

All works

10 works of theirs in our library, the most cited first; 5 with a page here, the others link their original until we summarise them.

1. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model ↗arXiv:2211.05100 · no summary yetBigScience Workshop et al.2022 · 1,606 citations
2. Holistic Evaluation of Language Models ↗arXiv:2211.09110 · Published in Transactions on Machine Learning Research (TMLR), 2023 · no summary yetPercy Liang et al.2022 · 1,001 citations
3. PipeDream: Fast and Efficient Pipeline Parallel DNN TrainingarXiv:1806.03377Aaron Harlap, Deepak Narayanan, Amar Phanishayee, et al.2018 · 162 citations
4. Memory-Efficient Pipeline-Parallel DNN TrainingInternational Conference on Machine Learning (ICML)Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, et al.2020 · 153 citations
5. Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning WorkloadsarXiv:2008.09213Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, et al.2020 · 141 citations
6. Holistic Evaluation of Text-to-Image Models ↗neural information processing systems · no summary yetTony Lee et al.2023 · 77 citations
7. An Empirical Study of Mamba-based Language ModelsarXiv:2406.07887Roger Waleffe, Wonmin Byeon, Duncan Riach, et al.2024 · 67 citations
8. Nemotron-4 15B Technical ReportarXiv:2402.16819Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, et al.2024 · 30 citations
9. Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs ↗arXiv (Cornell University) · no summary yetDeepak Narayanan et al.2023 · 6 citations
10. Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer Models ↗neural information processing systems · no summary yetDeepak Narayanan et al.2023 · 3 citations

🤝 Co-authors here

12 of 18 co-authors with a page, the most papers together first.

• Amar Phanishayee ×3
• Matei Zaharia ×2
• Albert Gu
• Ayush Dattagupta
• Brandon Norick
• Chen Zhu
• Dan Su
• Joseph Jennings
• Jupinder Parmar
• Keshav Santhanam
• Mostofa Patwary
• Sandeep Subramanian

Venues

Where their works here were published, the most works first, with the editions of those that have a page.

• neural information processing systems ×2
• International Conference on Machine Learning (ICML): ICML 2020

🔗 Cited by

Papers of our library cite their works 3,829 times (of the 149,439 papers whose bibliographies we read); 802 citing papers with a page here →.

Updated 2 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/authors/deepak-narayanan/