Why doesn't a 1.03–1.16× from choosing CUTLASS configurations per shape count as a speedup yet?

We started with a fairly ordinary question. On an RTX 5090, a block-scaled FP4 (NVFP4) matrix multiply usually goes through torch._scaled_mm to cuBLAS and its default heuristic. Could we do better on the shapes an LLM actually runs by choosing a CUTLASS configuration for each shape?

Our first result looked encouraging. On one card, in one fixed-order run, top-dispatcher was faster on all eight shapes: 1.03× to 1.16×, at 1,146–1,338 TFLOPS.

Then we looked at the measurement order. We had always timed our kernel first. After the power and frequency results in the previous note, that was hard to overlook. We still have a candidate worth testing, but we cannot call those ratios a speedup yet.

What did we actually change?

We did not invent a new kernel. The tiles, schedules and epilogue are CUTLASS 4.5 collectives. Our contribution here is the choice of which of three configurations to launch for a given shape.

We swept 25 shapes across 15 choices: five configurations, each at three swizzles. Selecting the lowest recorded ncu duration for each shape gave 1.024–1.261× relative to the 128×128×128, swizzle-1 default. Selecting the highest tensor-pipe utilization did not always select that fastest row; its range includes 1.000×. These are in-sample selections from one historical sweep, not a held-out result for the dispatcher.

The table ended up fairly small. One 256×128×128 tile won most shapes. Under the minimum-duration selector, ping-pong only won the square 4K and 8K×4K×4K cases; stream-K won when K was long relative to N. Our working explanation is that, with few output tiles, splitting the K loop keeps more of the 170 SMs busy. We have not profiled that explanation yet.

We can also pair eleven shapes with the historical b12x table. For this comparison we use the maximum-utilization choice from our sweep and compare the recorded ncu durations. The ratio ranges from 0.987× to 1.218×, including one shape where b12x is faster. This retrospectively selected arm is not a measurement of the delivered dispatcher against b12x.

Do the outputs agree?

We checked random E2M1 data, random UE4M3 scales in [0.5, 2), and alpha 0.75. Across five shapes, including M = 1000, which is not a multiple of the tile, the largest absolute difference from torch._scaled_mm, divided by the largest absolute output, was 3.65×10⁻³ to 4.20×10⁻³.

That result was identical for all three configurations and is consistent with BF16 output rounding. It is also a limited check: a matrix-wide maximum can hide relative error near zero. We still need an error distribution against an independent dequantized reference.

Where would this help in serving?

We have not tuned decode. The dispatcher works below M = 256, but nothing there is tuned, and decode is where a 5090 serving an LLM spends most of its steps.

We also have a reason to be cautious about isolated GEMM wins. A different MoE kernel of ours was 1.20× faster per layer in a microbenchmark, then only 1.04–1.11× faster on the real prefill path. We stopped that line of work. This dispatcher has not established an end-to-end speedup either.

What would make us trust the 1.03–1.16×?

The original test timed top-dispatcher first and torch._scaled_mm immediately after. The baseline always ran later in each pair, creating a potential thermal and power-order bias in our favour; its size is unknown.

Each number comes from one sequence (n = 1) of 50 calls, or 20 for the two largest shapes, after 3 warm-up calls. Inputs were random and clocks were unlocked. The dispatch table was also tuned on overlapping shapes.

Before publishing a speedup claim, we will rerun the eight shapes in balanced ABBA/BAAB pairs, logging clock and power per block. We will add a tuned cuBLASLt arm, time pre-packed GEMM and the full call separately, and test shapes the table never saw.

For now, these are candidate results awaiting a retest. If the 3% points fall inside the run-to-run spread, we will say so.