Operator Explainers (2): Model Quantization for Deployment — the precision-compression spectrum from FP16 to FP4

0. The One-Line Thread

Model quantization is the art of trading precision for memory/bandwidth/compute — it is orthogonal to distillation (which trades capacity for size); the two stack (distill a small model, then quantize it for deployment). This post consolidates the FP8/INT4/FP4/KV-Cache quantization notes that were scattered across daily tracking into one spectrum, landing on real vLLM/SGLang and speculative-decoding deployments.

1. Precision Spectrum and Compression

• BF16 vs FP16: BF16 has more exponent bits (stable training range); FP16 has finer mantissa but overflows easily.
• Two FP8 formats: E4M3 (4 exp / 3 mant, good for weights/acts, code-math friendly) vs E5M2 (5 exp / 2 mant, larger range, good for long-text/multilingual) — the choice alone moves the accuracy drop.

2. Post-Training Quantization (PTQ): compression without gradients

PTQ quantizes a trained model directly; the core question is “how to assign scaling factors”:

• RTN (round-to-nearest): per-element round; fastest but large-activation channels get crushed.
• GPTQ: per-layer OBS-style second-order quantization — when quantizing one column, compensate the others via the Hessian, minimizing ‖WX − ŴX‖². Good for offline GPU layer-by-layer compression.
• AWQ (activation-aware): compute activation saliency s = mean(|X|)^α, “amplify” important channels before quantizing; near-zero overhead, excellent fidelity on large-activation channels. The current LLM on-device default.
• per-group: group_size=128 is the industry default granularity, balancing fidelity and lookup-table cost.

Rotation-based quantization (outlier suppression) — the frontier branch of PTQ. Its mathematical lever is the rotational invariance of linear maps: for any orthogonal R (R⁻¹=Rᵀ), XW = (XR⁻¹)(RW). Left-multiplying weights by R and right-multiplying activations by Rᵀ leaves the product unchanged but “stirs” the numerical distribution — the activation outliers concentrated in a few channels get flattened, so 4-bit quantization no longer overflows its dynamic range. That is the key that makes W4A4 viable.

• QuaRot (NeurIPS 2024): uses a randomized Hadamard transform to build the orthogonal rotation, applying R1–R4 rotations across weights, activations, and KV Cache simultaneously — the first to hit W4A4KV4 (4-bit weights + 4-bit acts + 4-bit KV) near-losslessly on LLaMA-2/3, training-free and plug-and-play.
• SpinQuant (arXiv:2405.16406): upgrades the rotation from “random fixed” to “learnable” — optimizing the rotation matrices on the Stiefel manifold via Cayley SGD so the rotation fits both weight and activation distributions. On LLaMA-3 it beats QuaRot across the board, recovering up to 45.1% more accuracy, and folds RoPE into the rotation optimization too.
• DuQuant: adds channel permutation on top of rotation, scattering outliers into adjacent channels for even steadier W4A4.

Deployment note: rotation is orthogonal to AWQ/GPTQ — AWQ protects salient channels, GPTQ does second-order compensation, the rotation camp flattens outliers; the three stack (rotate then AWQ), and that stacking is the key piece for pushing 4-bit on-device inference to usable today.

VLA in practice: OpenVLA INT4 ≈ 4GB with near-lossless accuracy runs on Jetson Orin, leaving headroom for perception/planning/control — the only way “big models enter the physical world.”

3. Quantization-Aware Training (QAT): put the error into backprop

PTQ loss grows at INT4/FP4, so QAT is needed:

• FP4 QAT (DeepSeek V4 / V4-Flash line): apply FP4 pseudo-quantization to MoE routed-expert weights and the CSA indexer QK path — pseudo-quantize to FP4 during training then dequantize back to high precision for the compute, while sampling/rollout uses native FP4. Gradients thus see the true post-quant distribution; deployment is WYSIWYG.
• Prerequisite: mixed-precision training (BF16 AMP) + FP8 KV Cache. QAT usually pairs with Muon optimizer and SwiGLU Clamping to keep 4-bit drops usable.

4. KV Cache quantization and eviction: the long-context memory lifeline

The main bottleneck of long-context inference is not FLOPs but KV Cache memory. Two orthogonal paths:

• Quantize (compress precision): FP16→FP8(E4M3/E5M2) 2× ~0.3%; FP16→INT4 mixed (K symmetric + V asymmetric) 4× ~0.5%; per-group(128) mainstream.
• Evict (cut token count): H2O / ThinKV prune by importance, another 3~244×.
• Frontier stacking: RDKV (2026) unifies quant+eviction under rate-distortion, 2.48% cache → 97.81% accuracy; ThinKV (ICLR 2026 Oral) thought-adaptive KV compress, <5% KV kept near-lossless, 5.8× throughput.
• Pitfalls: E4M3 vs E5M2 choice, INT2 practical floor, quant+evict tuning, H2O’s extra 5~10% compute.

5. Landing on inference engines and speculative decoding

• vLLM / SGLang: FP8 KV Cache + AWQ deployment is where quantization meets the engine; HiSparse (see sys6) pushes “HBM as cache, host DRAM as authoritative KV” — a system-level extension of the quant+tier idea.
• Speculative decoding: quantization frees KV headroom — bigger batch, heavier draft model, more end-to-end speedup.
• Training side: Megatron-LM --fp8-format hybrid + BF16 AMP is now standard for large-model training.

6. Deployment checklist

1. Locate the bottleneck first: memory wall → KV quant/evict; bandwidth wall → INT4/AWQ weights; compute wall → FP8 forward.
2. Prefer FP8 (2×, ~0.3%) ; drop to INT4 (AWQ, 4×) only on-device.
3. FP4 must come with QAT, or only for drop-insensitive scenarios.
4. Long text → E5M2; code/math → E4M3.
5. Below INT2 is basically unusable — don’t trade usability for a headline number.

Tying it together: distillation shrinks capacity, quantization compresses precision, KV eviction cuts length, tiered caching (HiSparse) puts all three into one memory budget — that is the full toolbox for squeezing 70B+ models into single-card long-context serving today.

Note: this post consolidates the 2026-07-18 (KV Cache quant+evict), 2026-08-01 (FP8/INT4 deploy), 2026-08-04 (FP4 QAT) daily technical tracks into a systematic version.