Elamur Research⌕
☰
home / papers / 2023 / Microscaling Data Formats for Deep Learning

Microscaling Data Formats for Deep Learning

Paper · 2023 · Bita Darvish Rouhani, Ritchie Zhao, Ankit More et al.

Summary

Low-precision number formats lower the compute and memory cost of deep learning. However, formats narrower than 8 bits cannot cover the value range of a whole tensor with a single scale factor. This paper, from an industry group including Microsoft, AMD, Intel, Meta, NVIDIA and Qualcomm, evaluates the Microscaling (MX) family from the open OCP standard. In MX, small blocks of 32 values share one power-of-two scale, and each value is stored as a tiny float or integer. The authors emulate MX on GPUs with a PyTorch CUDA extension. They test four settings across more than two dozen benchmarks in translation, text encoding, vision, speech, recommendation and LLMs, including GPT3-175B and LLaMA-7B:

  • direct casting of FP32 models
  • a post-training quantization method
  • quantization-aware finetuning
  • training from scratch

MXINT8 works as a drop-in substitute for FP32 inference. MXFP6 nearly matches FP32 after finetuning. GPT-style models trained entirely in 6-bit MX reach FP32 loss with unchanged hyperparameters, which the authors present as the first such sub-8-bit training result. This matters for designing cheaper training and inference hardware.

Key points

  • Formats: all concrete formats use blocks of 32 elements and an 8-bit exponent-only scale (E8M0). The element types are:
    • MXFP8 (E4M3/E5M2)
    • MXFP6 (E2M3/E3M2)
    • MXFP4 (E2M1)
    • MXINT8
  • Compute flow: matmuls and convolutions in both forward and backward passes take MX inputs. Normalization, Softmax, activations and residual adds run in Bfloat16 or FP32, and an FP32 master copy of the weights is updated.
  • Direct-cast inference: for MXFP8 and MXFP6, the variant with more mantissa bits did better. On LLaMA-7B, wikitext perplexity is:
    • 9.488 for FP32
    • 9.504 for MXINT8
    • 27.201 when everything is MXFP4
  • Training GPT models (20M–1.5B parameters):
    • MXFP6_E3M2 for weights, activations and gradients gives loss nearly equal to FP32 (2.75 vs 2.74 at 1.5B).
    • MXFP4 weights with MXFP6 activations and gradients give 2.76.
  • Post-training quantization: the PTQ method ("error diffusion") is derived from GPFQ. It makes one pass over a small calibration set and needs no tuning parameters.

Interesting ideas

  • I1. Offset the shared exponent: the block's shared exponent is set from the block's largest magnitude, then shifted so that this value lands in the element format's top binade. This uses the full range of a tiny format, and the same rule could serve any block-wise quantizer, such as embedding or KV-cache compression.
  • I2. Separate scale from element type: scale format, element format and block size are defined independently, so one framework covers many bit-widths. A modular spec like this makes extending or comparing formats straightforward.
  • I3. Plan for transposes: quantizing and transposing do not commute, because the scale is tied to one axis. Training code must therefore keep separately quantized copies of a weight and its transpose, a subtle point for anyone implementing block-scaled kernels.
  • I4. Reuse the FP32 recipe: judging a format by whether it trains with unchanged FP32 hyperparameters directly measures "user friction." This is a practical evaluation criterion beyond plain accuracy.
  • I5. Grade by adaptation effort: comparing direct-cast, PTQ, finetuning and full training shows how much effort each format needs. This tiered protocol is useful for evaluating any compression scheme.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksComputational Theory and Mathematics · 321 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

Cite

Cite as Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger and Kristof Denolf. 2023. Microscaling Data Formats for Deep Learning. arXiv 2023 (OCP MX consortium).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2310.10537

@misc{rouhani2023microscaling,
  title = {{Microscaling Data Formats for Deep Learning}},
  author = {Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger and Kristof Denolf},
  year = {2023},
  eprint = {2310.10537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.10537}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paulius Micikevicius, Stuart Oberman, Pradeep Dubey, Marius Cornea, Andres Rodriguez, Ian Bratt, Richard Grisenthwaite, Norm Jouppi, Chiachen Chou, Amber Huffman, Michael Schulte, Ralph Wittig, Dharmesh Jani, and Summer Deng. OCP 8-bit Floating Point Specification (OFP8). Open Compute Project, 2023. find
  2. Mario Drumond, Tao Lin, Martin Jaggi, and Babak Falsafi. Training DNNs with Hybrid Block Floating Point. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018. find
  3. Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, XIA SONG, Subhojit Som, Kaustav Das, Saurabh T, Steve Reinhardt,… find
  4. Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany. VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference. Machine Learning and Systems (MLSys, 3:873–884, 2021. find
  5. Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, Lei Shao, Gaurav Kolhe, Dimitry Melts, Jasmine Klar, Renee L’Heureux, Matt Perry, Doug Burger, and Eric Chung. With Shared Microexp… find
  6. Bita Darvish Rouhani, Nitin Garegrat, Tom Savell, Ankit More, Kyung-Nam Han, Mathew Zhao, Ritchie amd Hall, Jasmine Klar, Eric Chung, Yuan Yu, Michael Schulte, Ralph Wittig, Ian Bratt, Nigel Stephens, Jelena Milanovic, John Brothers, Pradeep Dubey, Marius Cornea, Alexander Heinecke, Andres Rodriguez… find
  7. Microscaling PyTorch Library. 2023. URL https://github.com/microsoft/microxcaling. ↗
  8. Jinjie Zhang, Yixuan Zhou, and Rayan Saab. Post-training quantization for neural networks with provable guarantees. arXiv:2201.11113, 2022. — [arXiv:2201.11113](https://arxiv.org/abs/2201.11113) find
  9. Transformer For PyTorch. URL https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/Transformer. ↗
  10. GNMT v2 For PyTorch. URL https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/GNMT. ↗
  11. NVIDIA/Megatron-LM: Ongoing research training transformer. URL https://github.com/NVIDIA/Megatron-LM. ↗
  12. Data-Efficient architectures and training for Image classification. facebookresearch/deit. find
  13. Convolutional Network for Image Classification in PyTorch. URL https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets. ↗
  14. Torchvision MobileNetV2. URL https://github.com/pytorch/vision. ↗
  15. wav2vec 2.0. URL https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec. ↗
  16. Deep Learning Recommendation Model for Personalization and Recommendation Systems. URL https://github.com/facebookresearch/dlrm. ↗
  17. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  18. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361, 2020. URL https://github.com/ ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rouhani-2023-microscaling-data-formats-deep-learning/