Elamur Research⌕
☰
home / papers / 2026 / Quantizing Whisper: How Design Choices Affect ASR Performance

Quantizing Whisper: How Design Choices Affect ASR Performance

Paper · 2026 · Arthur Söhler, Julian Irigoyen and Andreas Søeborg Kirkedal

Summary

Whisper models transcribe speech accurately but need a lot of memory and compute, which makes them hard to run on edge devices or in low-resource settings. The authors ask how practical post-training quantization (PTQ) choices affect Whisper's accuracy, speed and size when no retraining is done. They quantize Whisper-small (244M parameters) with four libraries: PyTorch, Optimum-Quanto, HQQ and bitsandbytes. The comparison varies the scheme (dynamic vs. static), the method (symmetric vs. asymmetric), the granularity (per-tensor vs. per-channel) and the bit-width, from int8 down to int3 and nf4. Every configuration is measured the same way on CPU and GPU using LibriSpeech test-clean and test-other. The best balance comes from dynamic int8 in Optimum-Quanto, which shrinks the model by roughly 57% and slightly improves WER over the unquantized baseline. Static quantization does worse, and very low-bit formats save more space but lose accuracy on harder audio. Repeating the tests on Whisper-base and Whisper-tiny gives the same overall picture. The study offers concrete guidance for deploying Whisper on limited hardware.

Key points

  • Setup: Whisper-small is evaluated with PyTorch, Optimum-Quanto, HQQ and bitsandbytes (BNB) at int8, int4, int3, nf4 and fp8, depending on what each library supports. Accuracy, latency and compression are measured on CPU and GPU.
  • Best configuration: Dynamic int8 in Optimum-Quanto cuts model size by about 57% and lowers WER below the full-precision baseline on LibriSpeech.
  • Static quantization: It underperforms dynamic quantization, probably because efficient low-bit kernels for LayerNorm and Softmax are missing.
  • Aggressive formats: nf4 and int3 reach up to about 71% compression, but accuracy drops, especially on the acoustically harder test-other set.
  • Other model sizes: Whisper-base and Whisper-tiny show the same trends, with sharper degradation at low bit-widths. Whisper-medium and Whisper-large were not tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Copenhagen Business SchoolCopenhagen · Denmark6 works · 53 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Speech & spoken language · 480 worksEfficient models & compression · 242 works

Cite

Cite as Arthur Söhler, Julian Irigoyen and Andreas Søeborg Kirkedal. 2026. Quantizing Whisper: How Design Choices Affect ASR Performance. Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4f2o9uqpxhn7

@inproceedings{sohler2026quantizing,
  title = {{Quantizing Whisper: How Design Choices Affect ASR Performance}},
  author = {Arthur Söhler, Julian Irigoyen and Andreas Søeborg Kirkedal},
  booktitle = {Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4f2o9uqpxhn7},
  url = {https://doi.org/10.63317/4f2o9uqpxhn7}
}

References

16 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allison Andreyev. 2025. Quantization for OpenAI’s Whisper Models: A Comparative Analysis. arXiv preprint arXiv:2503.09905. — [arXiv:2503.09905](https://arxiv.org/abs/2503.09905) find
  2. Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. 2021. A White Paper on Neural Network Quantization. arXiv preprint arXiv:2106.08295. — [arXiv:2106.08295](https://arxiv.org/abs/2106.08295) find
  3. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. GPT3.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In Advances in Neural Information Processing Systems, volume 35, pages 30318–30332. Curran Associates, Inc. find
  4. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: an ASR corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pages 5206–5210. IEEE. find
  5. Chen Feng, Yicheng Lin, Shaojie Zhuo, Chenzheng Su, Ramchalam Kinattinkara Ramakrishnan, Zhaocong Yuan, and Xiaopeng Zhang. 2025. Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models. arXiv preprint arXiv:2507.07877. — [arXiv:2507.07877](https://arxiv.org/abs/2507.07877) find
  6. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–2…
  7. Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021. A Survey of Quantization Methods for Efficient Neural Network Inference. arXiv preprint arXiv:2103.13630. — [arXiv:2103.13630](https://arxiv.org/abs/2103.13630) find
  8. R.M. Gray and D.L. Neuhoff. 1998. Quantization. IEEE Transactions on Information Theory, 44(6):2325–2383. find
  9. Haihao Shen, Naveen Mellempudi, Xin He, Qun Gao, Chang Wang, and Mengni Wang. 2024. find
  10. Efficient Post-training Quantization with FP8 Formats. In Proceedings of Machine Learning and Systems, volume 6, pages 483–498. find
  11. Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao. 2011. Improving the speed of neural networks on CPUs. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011. find
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  13. Lu Wei, Zhong Ma, Chaojie Yang, and Qin Yao. 2024. Advances in the Neural Network Quantization: A Comprehensive Review. Applied Sciences, 14(17). find
  14. Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. 2020. Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation. arXiv preprint arXiv:2004.09602. find
  15. Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. In Advances in Neural Information Processing Systems, volume 35, pages 27168– 27183. Curran Associates, Inc. 1…
  16. Language Resource References Panayotov, Vassil and Povey, Daniel. 2015. LibriSpeech ASR corpus. OpenSLR. PID https://www.openslr.org/12. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sohler-2026-quantizing-whisper-how-design-choices/