Decomposing MXFP4 quantization error for LLM RL: reducible bias, recoverable deadzone, and an irreducible floor

TL;DR

MXFP4 promises 4× throughput and 4× memory for LLM training on Blackwell/MI350-class accelerators, but naively dropping it into RL post-training destroys accuracy. Existing work treats the quantization error as one monolithic noise term, which makes it impossible to explain why the same MXFP4 hurts dense and MoE models in completely different ways.

We prove that the MXFP4 error is exactly the sum of three components with disjoint structure:

• Scale bias from the E8M0 power-of-two scale → corrupts the gradient
• Deadzone truncation from zeroing small values → corrupts the rollout
• Grid noise from rounding to the E2M1 grid → raises policy entropy

Each correction targets one mechanism: MBS (macro-block scaling) kills the scale bias, OF (outlier fallback with blend $\alpha=0.5$) recovers the deadzone, and AQN controls the residual grid noise. Validated on Qwen2.5-3B and Qwen3-30B-A3B-Base:

• Dense: recovers BF16 within $-0.7$ pp (81.3% vs. 82.0%)
• MoE: AQN+MBS+OF ($\alpha=0.5$) exceeds BF16 by $+1.0$ pp (92.49% vs. 91.51%)

The Problem: One Monolithic Noise Term Hides Three Mechanisms

The MXFP4 format (OCP MX standard) stores each block of 32 elements as:

1. A shared E8M0 scale — 0 mantissa bits, only powers of two
2. E2M1 per-element values on the grid $\mathcal{G} = \{0, \pm 0.5, \pm 1, \pm 1.5, \pm 2, \pm 3, \pm 4, \pm 6\}$

Replace BF16 with this in RL post-training and accuracy collapses:

• Qwen2.5-3B (dense): −21.9 pp on GSM8K
• Qwen3-30B-A3B-Base (MoE): −2.7 pp

The standard PTQ/QAT literature (GPTQ, AWQ, QuIP, QeRL, …) treats the quantization error $e = Q(x) - x$ as a single noise term and patches the symptoms. That hides the structural question: why does dense lose 21.9 pp but MoE only 2.7 pp? Different mechanisms must be dominant in each architecture.

The Key Insight: Three Disjoint Error Components

Introduce an ideal-scale quantizer $Q^* $ that keeps the unquantized scale $s_b^* = \max_i |x_{b,i}| / q_{\max}$ instead of E8M0’s ceiling-rounded $s_b$. Then the total error decomposes exactly:

$$e_{b,i} = \underbrace{Q(x_{b,i}) - Q^*(x_{b,i})} _{e^{\mathrm{scale}}_{b,i} \text{ (scale bias)}} + \underbrace{[Q^*(x_{b,i}) - x_{b,i}] \cdot \mathbb{1}_{\mathcal{D}_b}(i)}_{e^{\mathrm{DZ}}_{b,i} \text{ (deadzone)}} + \underbrace{[Q^*(x_{b,i}) - x_{b,i}] \cdot \mathbb{1}_{\mathcal{D}_b^c}(i)}_{e^{\mathrm{grid}}_{b,i} \text{ (grid noise)}}$$

where $\mathcal{D}_b$ is the deadzone (elements with $|x_{b,i}| < m_b / 24$, mapped to 0).

This decomposition has two formal properties that drive everything that follows.

Property 1: Exact Orthogonality

Lemma (pointwise, no distributional assumptions):

$$\langle \mathbf{e}^{\mathrm{DZ}}, \mathbf{e}^{\mathrm{scale}} \rangle = \langle \mathbf{e}^{\mathrm{DZ}}, \mathbf{e}^{\mathrm{grid}} \rangle = 0$$

Proof sketch: On the deadzone, $|x/s_b^*| < q_{\min}/2$, so $Q^*(x) = 0$. Ceiling rounding guarantees $s_b \geq s_b^*$, so $|x/s_b| \leq |x/s_b^*| < q_{\min}/2$ and $Q(x) = 0$ as well. Hence $e^{\mathrm{scale}} = Q - Q^* = 0$ identically on the deadzone. $\square$

The MSE identity then has just one surviving cross term:

$$ \|\mathbf{e}\|^2 = \|\mathbf{e}^{\mathrm{scale}}\|^2 + \|\mathbf{e}^{\mathrm{DZ}}\|^2 + \|\mathbf{e}^{\mathrm{grid}}\|^2 + 2 \langle \mathbf{e}^{\mathrm{scale}}, \mathbf{e}^{\mathrm{grid}} \rangle $$

Empirically that cross term has $\cos(\mathbf{e}^{\mathrm{scale}}, \mathbf{e}^{\mathrm{grid}}) \approx -0.66$ across 18,876 weight tensors in two model scales — a structural anti-correlation from the ceiling property $s_b \geq s_b^*$.

Property 2: Grid Noise Is Scale-Invariant ⇒ Irreducible Floor

The grid error $e^{\mathrm{grid}}$ depends on the weights and the ideal scale $s_b^*$, not on the actual E8M0 scale $s_b$. As you improve scale precision (E8M0 → E8M$k$):

$$ \|\mathbf{e}^{\mathrm{scale}}\|^2 \to 0, \quad \langle \mathbf{e}^{\mathrm{scale}}, \mathbf{e}^{\mathrm{grid}} \rangle \to 0, \quad \|\mathbf{e}\|^2 \to \underbrace{\|\mathbf{e}^{\mathrm{grid}}\|^2 + \|\mathbf{e}^{\mathrm{DZ}}\|^2}_{\text{irreducible floor}} $$

Total error converges to a floor set by the E2M1 grid itself. The only way to go below it is to change the grid (e.g., NVFP4’s larger format) or to recover deadzone values directly (outlier fallback).

The ratios are remarkably stable across model scales:

These numbers are properties of the MXFP4 format itself, not of any particular model.

Each Component Dominates One RL Failure Mode

The decomposition matters because each component breaks RL training through a different mechanism.

Scale bias → gradient accuracy (backward-pass-only pathology)

E8M0 ceiling rounding gives a per-block scale ratio $\gamma_b = s_b / s_b^* = 2^{\delta_b}$ with $\delta_b \sim \mathrm{Uniform}[0, 1)$, i.e. a $\sim$44% systematic bias per layer. In the forward pass, LayerNorm resets activations every layer, preventing accumulation. But in the backward pass, the STE chain rule multiplies $L$ scale factors with no normalization:

$$ \log \frac{\|\hat{\nabla}\|}{\|\nabla_{\mathrm{true}}\|} \approx \sum_{l=1}^L \delta_{b(l)}, \qquad \delta_b \sim \mathrm{Uniform}[0, 1) $$

For 48 layers, the centred sum has std $\sqrt{L/12} = 2.0$. Within one standard deviation the gradient magnitude ratio spans roughly $[0.18\times, 5.6\times]$. We measure $\sigma_{\mathrm{emp}} = 1.97$ on Qwen3-30B-A3B-Base, in tight agreement with theory.

Deadzone → rollout quality (forward-pass-only pathology)

Deadzone is information loss: about 9% of weights are mapped to zero. The forward pass loses rank, producing blander rollouts. But the STE backward pass ignores the deadzone — it passes gradients through as if the full-precision value were present. So deadzone damage is fully visible in rollouts but largely invisible in gradients.

This asymmetry is what makes deadzone primarily a rollout quality problem, not a gradient problem.

Grid noise → policy entropy (effective temperature)

Grid noise is approximately zero-mean Gaussian on logits, $\boldsymbol{\eta} \sim \mathcal{N}(0, \sigma_\eta^2 \mathbf{I})$. By matching the noisy softmax to a deterministic temperature-scaled softmax via the classical probit–logit identity (convolving a probit with a Gaussian gives another probit with summed variances),

$$ T_{\mathrm{eff}} \approx \sqrt{1 + \dfrac{2 \sigma_\eta^2}{\mathrm{Var}(\Delta \ell)}} \; > \; 1 $$

The factor of 2 comes from $\mathrm{Var}(\eta_a - \eta_b) = 2\sigma_\eta^2$ — the variance of a pairwise logit difference. The noise level is constant throughout training, so it provides no annealing mechanism — exploration is widened uniformly. This causes premature entropy collapse if not actively managed.

Three Mechanism-Targeted Corrections

MBS — Macro Block Scaling

E8M0 wastes half a magnitude because it has zero mantissa bits. MBS adds an 8-bit mantissa correction at a coarser macro-block granularity ($B_M = 128$, four MXFP4 blocks):

$$ s^{\mathrm{MBS}} = 2^{e_M} \cdot (1 + m_{\mathrm{MBS}}), \qquad m_{\mathrm{MBS}} \in [0, 1) $$

Implemented in training as a prescale–quantize–postscale wrapper:

$$ \hat{x}_i = \frac{1}{1 + m_{\mathrm{MBS}}} \cdot Q\!\bigl((1 + m_{\mathrm{MBS}}) \cdot x_i\bigr) $$

Cost: 1 byte / 128 elements ≈ <0.1 bit/element, with prescale/postscale folded into the GEMM epilogue at near-zero compute cost. Effect: $\mathrm{Var}(\gamma)$ shrinks by $\sim (256)^{-2}$, the scale component vanishes, and the cross term with grid noise vanishes too — total MSE converges to the irreducible floor.

OF — Outlier Fallback (with residual blend α)

Deadzone is information loss; no rescaling recovers it. OF uses a two-pass residual quantization:

$$ \hat{\mathbf{x}}_1 = Q(\mathbf{x}), \quad \hat{\mathbf{x}}_2 = Q(\mathbf{x} - \hat{\mathbf{x}}_1), \quad \hat{\mathbf{x}}_{\mathrm{OF}} = \hat{\mathbf{x}}_1 + \alpha \, \hat{\mathbf{x}}_2 $$

Pass 1 quantizes outliers accurately (they set the block scale) and sends small values to the deadzone. Pass 2 has small dynamic range, so previously dead values are now representable on the E2M1 grid.

We introduce the residual-blend coefficient $\alpha$. Setting $\alpha = 1$ adds the full residual back, but $\hat{\mathbf{x}}_2$ is itself an MXFP4-quantized approximation and inherits its own grid+deadzone error. Adding it at half strength ($\alpha = 0.5$) outperforms $\alpha = 1$ on GSM8K by ~1 pp:

There’s also a training-dynamics reason for $\alpha = 0.5$. In a controlled single-step isolation run (OF only, MBS/AQN disabled, learning rate 0, fixed prompts so $\alpha$ is the only varying quantity), the rollout-vs-training numerical drift is U-shaped in $\alpha$ with a clean minimum at $0.5$ — both extremes ($\alpha=0$ and $\alpha=1$) incur ${\sim}1.5$–$2\times$ the drift. A half-strength residual best aligns the rollout fake-quantization grid with the training grid; because RL post-training is drift-sensitive, the $\alpha$ that minimizes drift is also the $\alpha$ that maximizes downstream accuracy.

AQN — Adaptive Quantization Noise

After MBS and OF, the residual error is dominated by grid noise — static and unannealed. AQN injects controlled Gaussian noise on weights before each rollout with an exponential decay schedule from $\sigma_{\mathrm{start}} = 1%$ to $\sigma_{\mathrm{end}} = 0.1%$. This converts the constant grid temperature into a annealable exploration signal.

A subtle but important point: AQN only works after MBS removes the first-moment scale bias. Injecting AQN on top of biased MXFP4 logits diverges — the systematic bias contaminates the noise. This is the difference between our W4A4 full-parameter setting and QeRL’s W4A16 + LoRA setting, where MBS isn’t necessary.

Experiments: Does the Recipe Actually Hold Together?

We evaluate on GSM8K with verifiable rewards, using GRPO + Truncated Importance Sampling. Models:

• Qwen2.5-3B dense (36 layers, FSDP2)
• Qwen3-30B-A3B-Base MoE (48 layers, 3B active, Megatron)

All experiments are W4A4 QDQ emulation — numerically faithful to native MXFP4 (same E2M1 grid, same E8M0 block scale, FP32 accumulation).

MoE results (Qwen3-30B-A3B-Base, BF16 = 91.51%)

AQN+MBS+OF ($\alpha = 0.5$) exceeds BF16 by +1.0 pp — the new best recipe.

Dense results (Qwen2.5-3B, BF16 = 82.0%)

Naive MXFP4 loses 21.9 pp; MBS+AQN+OF recovers to within 0.7 pp. Notably, OF alone provides +17.5 pp on dense but only +1.5 pp on MoE.

The dense vs. MoE dichotomy is predicted by the decomposition

Why does OF give $+17.5$ pp on dense but only $+1.5$ pp on MoE? Expert routing in MoE acts as a natural error-correcting code: if one expert’s small weights are pruned, other experts compensate. Dense models have no such redundancy, so the deadzone severs the forward path directly.

This is exactly the prediction of the decomposition — deadzone is a forward-pass problem, and forward-pass redundancy modulates its severity. Without this view, the asymmetry looks arbitrary.

Training dynamics: AQN prevents entropy collapse

Scope: the floor that averages out in short responses compounds in long ones

The “grid noise averages out over full responses” argument holds for short-to-medium responses (GSM8K, ~280 tokens / 1024-token budget). It reaches its limit as responses lengthen: holding the recipe family fixed, per-step rollout–training Pearson correlation is essentially identical on a short task (GSM8K) and a long one (DAPO-MATH, ~795 tokens) — yet outcomes diverge sharply, and on multi-thousand-token CoT the same recipe under-recovers substantially. The mechanism is an autoregressive trajectory fork under deterministic greedy decoding: a single early arg-max flip from quantization noise compounds into a different solution path. We treat long-CoT fidelity as scope-limited future work for this recipe (likely needing higher activation precision, e.g. W4A8).

Takeaways

1. MXFP4 quantization error is not one noise term. It is the exact sum of scale bias, deadzone, and grid noise — three components with disjoint structure, two formal properties (orthogonality + grid invariance), and consistent ratios across model scales.
2. Each component breaks a different part of RL training. Scale bias compounds exponentially in the backward pass but is forward-invariant; deadzone is the opposite (forward-fatal, gradient-invisible); grid noise raises effective temperature uniformly.
3. Corrections must be mechanism-targeted, not symptom-targeted. MBS removes scale bias, OF recovers the deadzone (with $\alpha = 0.5$ outperforming the naive $\alpha = 1$ by ~1 pp), AQN controls residual grid noise. The recipe exceeds BF16 on MoE and matches it within 0.7 pp on dense.
4. The dense/MoE dichotomy is no longer mysterious. Deadzone is a forward-pass problem; MoE expert routing provides forward-pass redundancy that masks it. Dense models cannot — so OF is the load-bearing correction on dense, while MBS+AQN is what carries MoE.

The broader principle: when accuracy collapses under aggressive quantization, decompose the error before patching symptoms. A single-noise view of MXFP4 hides exactly the structure that points to a working recipe.

References & Code

If you find our work useful, please kindly cite our work:

@misc{li2026decomposingmxfp4quantizationerror,
      title={Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor}, 
      author={Xiaocan Li and Shiliang Wu and Hei Yi Mak and Mehran Taghian and Yunke Peng and Zheng Shen},
      year={2026},
      eprint={2605.20402},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.20402}, 
}

Questions, comments, or want to try the recipe on your own MXFP4 stack? Feel free to reach out.