MTP made a single request faster. What happens when more requests arrive?

You enable multi-token prediction, run a request, and watch the tokens arrive faster. For someone waiting on that response, the improvement is straightforward.

Then you increase the load. More requests arrive, batches change, and memory fills up. Does the single-request speedup still translate into a better service?

In an earlier KVTop test series, the benefit at low concurrency did not persist as concurrency increased. That result applied to the model, implementation, workload, and memory budget we tested. It raised a practical question: where does MTP help, and what changes as the server gets busier?

What made the first request faster?

In the decoding approach considered here, MTP proposes multiple candidate tokens for verification. When enough candidates are accepted and verification is efficient, the request can advance several tokens with fewer sequential decoding steps.

The gain depends on how much useful progress each iteration makes relative to its cost. Drafting takes time, verification takes time, and rejected candidates represent work that does not become accepted output.

For a single request, we can measure that trade-off directly: how long does it take to produce the accepted output tokens?

What changes when those requests share a busy server?

The same work now competes for resources with other requests.

Drafting and verification consume compute. Depending on the implementation, additional state or memory reservations can also affect how many requests fit and how the runtime batches them.

This creates several possible outcomes. MTP might preserve its advantage as load rises. Its benefit might shrink as verification becomes more expensive. Or changes in memory availability and admission might leave more requests waiting.

The following curves compare deployment bundles in a historical closed-loop concurrency sweep; they do not isolate MTP or measure a controlled request-arrival rate.

The crossover itself does not tell us which explanation is correct. We need to inspect the execution and resource behavior around it.

In particular, submitting the same number of requests to both configurations does not guarantee that the server is executing the same number at once. Some may be active while others are still waiting for admission.

If the acceptance rate is high, why might the service still gain little?

Acceptance rate tells us how much proposed work survives verification. It does not tell us the full cost of producing that work.

Two runs can have similar acceptance rates and different drafting times, verification costs, or batch shapes. They can also leave different amounts of capacity available for other requests.

A useful comparison therefore tracks accepted tokens per iteration alongside iteration time. At the service level, it also follows requests through admission, execution, and completion.

That connects the local decoding gain to what users experience: how soon their response starts, how steadily it arrives, and how long it takes to finish.

How do we find the load range where MTP helps?

Keep the model, weights, quality criteria, GPU topology, and request-length distribution fixed. Compare MTP on and off while gradually increasing the request arrival rate.

At each load level, record actual active concurrency as well as queued requests. Track drafting and verification time, memory use, time to first token, and inter-token latency.

Then measure how much work completes within the chosen latency targets. State whether that means completed requests or their output tokens, and report failures and cancellations separately so they cannot disappear behind a throughput average.

If the curves diverge, investigate what changed at that point. A rise in verification cost calls for a different experiment from a drop in admitted concurrency. Traces and controlled comparisons are what let us move from observing the crossover to explaining it.

Should MTP be enabled by default?

The answer should follow the workload we expect to serve.

A lightly loaded interactive deployment may benefit from a setting that becomes less attractive under sustained load. Both results are useful, provided we report the conditions and verify that output quality remains acceptable.

The single-request test gives us the first answer. The load sweep tells us how far that answer carries.

Before choosing a default, we want to know where MTP improves the service—and what happens when traffic moves beyond that range.