KVTop
Serving · Long context · KV-cache admission

You configured a 1M context window. Can your server actually handle a 1M-token request?

KVTop · 2026-10-23 · 3 min read
≈ 493Kdefault packageadmission ceiling from the pool arithmetic (49,344 × 10), not a boundary scan
admitted twice1M requestTTFT 245 s and 202 s, on two different builds
n = 1evidenceeach 1M run once; raw run output re-read, not re-run

You’ve set the maximum context length to 1M. The server starts, the model loads, and shorter requests work.

Then you send a request near that limit. Can it get in? And will the answer change if other requests are already running?

The setting tells the runtime how long a sequence is allowed to be. Whether the server has enough resources to accept that sequence is a separate question.

What does changing the context limit actually change?

Raising the limit allows the runtime to consider longer sequences. Serving them still requires memory for their state, alongside model weights, execution workspaces, and other allocations.

Historical notes describe attempts near the configured 1M limit and a lower calculated capacity in a different default package. The available records do not establish a comparable admission boundary or completed-response result.

To understand that difference, it helps to separate three questions:

  • Configured limit: How long a sequence does the runtime allow?
  • KV-cache capacity: How much state can it store with the available memory budget?
  • Admission: Given the requests already running and the allocation policy, can it accept this one?

A request can be within the configured limit and still exceed what the server can accommodate.

Figure 1 · illustration, not data
SCHEMATIC · constraints, not measured capacitiesConfigured limitDoes the sequence fit the configured limit?Input and output accounting follows the runtime.Usable KV capacityCan every required stage store its share of the state?Weights, workspaces and other reservations reduce the budget.Admission nowCan this request enter under current load and policy?Active requests, reservations and allocation rules matter.All constraints matter; a larger configured limit does not create KV memory.
Why can a request fit the context limit but still not enter? Illustration only. Configuration, per-stage usable KV capacity, and admission policy are different constraints. Their units and allocation rules need not be interchangeable; no capacity or admission result is plotted.
scope
Conceptual resource checks; not a literal scheduler execution order
scale
No numeric scale, capacities or workload measurements

Source: Explanatory illustration for the section: What does changing the context limit actually change? and There’s enough VRAM across the GPUs. Why might the request still not fit?

There’s enough VRAM across the GPUs. Why might the request still not fit?

The total memory figure is only a starting point. What matters is how that memory is allocated and where the request needs it.

Weights and workspaces consume part of the budget. Other requests occupy cache capacity. With pipeline parallelism, each stage has its own constraints: one stage can run short even when there is spare memory elsewhere in the machine.

Figure 2 · illustration, not data
Can spare memory elsewhere rescue a full pipeline stage?SCHEMATIC · illustrative stages, not the tested topology; not to scaleStage ALocal weights / workspaceLocal KV allocationRequest needs this stageFitsStage BLocal weights / workspaceLocal KV allocationRequest needs this stageDoes not fitStage CLocal weights / workspaceLocal KV allocationRequest needs this stageFitsThe request needs an allocation at every required stage.Spare capacity at another stage does not automatically become usable here.
Can spare memory elsewhere rescue a full pipeline stage? Illustration only. A request must satisfy the allocation requirements at every participating pipeline stage. Spare memory elsewhere is not automatically transferable KV capacity; changing partitioning or placement is a separate configuration change.
scope
Conceptual illustration only; no measured memory amounts, durations or rates
scale
Not to scale; example stages are not the tested topology

Source: Supplied F2 draft: There’s enough VRAM across the GPUs. Why might the request still not fit?

That is why adding up the VRAM across all GPUs can give an overly optimistic picture. The request has to fit the actual layout and allocation rules.

When it is rejected, those budgets are worth inspecting before changing the maximum-context setting again.

What would count as a useful first test?

Start with a cold request, without relying on a reusable cached prefix. State exactly how many input tokens it contains and how much room is available for output.

That accounting matters near the limit. If the runtime counts input and output toward the same context window, filling it entirely with input leaves no room for an answer.

Figure 3 · illustration, not data
If the prompt fills the window, where does the answer go?SCHEMATIC · conditional on input and output sharing one context limitWithin the configured sequence limitInput tokenscounted by the runtimeOutput allowanceroom for generationInput accounting + allowed output must fit the configured limit.Include templates and special tokens when the runtime counts them.Actual KV reservation may be upfront or incremental, depending on policy.A sequence-length check alone does not prove enough memory is available.
If the prompt fills the window, where does the answer go? Illustration only. For runtimes that count prompt and generated tokens against one sequence limit, the input must leave room for output. Segment widths are arbitrary. Output allowance is not a claim that the runtime reserves all corresponding KV memory at admission.
scope
Conceptual illustration only; no measured memory amounts, durations or rates
scale
Not to scale; example stages are not the tested topology

Source: Supplied F2 draft: What would count as a useful first test?

Then follow the request through three events: admission, first token, and completion.

Acceptance establishes that the request passed the initial checks. Completion shows that the server carried it through execution. Time to first token tells us how long the user waited before the response began. Each observation answers a different part of the deployment question.

Figure 4 · measured
Run / attempt1M-token requestPrefill KV capacityBackground and resultFirst tokenEnd to end
Run A (engine 71ef9c61)1,040,000 tokens sent; decode side logged 1,040,0321,085,568 tokens (ratio 22)HTTP 200; the same run’s c = 32 background arm has no data after a decode-side crash244.96 s246.21 s
Run B (engine 6edc57f4), first injection1,040,000 tokens sent2,269,824 tokens (ratio 46)HTTP 200 while serving c = 32 background load (background TTFT p50 19.9 s, p99 183 s)202.08 s203.41 s
Run B, second injection1,040,000 tokens sentsame runPrefill bootstrap failed; timed out at 300 s——
Did the extra capacity turn into a completed long request? Each row is one historical injection (n = 1 per attempt). The two HTTP 200 responses came from different engine revisions and pool ratios; a second injection in Run B timed out. Output was capped at 32 tokens; actual generated length and finish reason were not recorded. The default package’s 49,344 × 10 = 493,440 tokens is configured pool capacity, not a measured admission boundary. These rows establish first-token delivery for the successful attempts, not sustained capacity or a measured limit.
model
GLM-5.3, 2-bit GGUF, prefill and decode disaggregated on RTX 5090 hosts
request
cold 1,040,000-token prompt, max 32 new tokens, ignore_eos
dates
2026-10-12 (A), 2026-10-13 (B)
repeats
n = 1 per run
finish_reason
not recorded; max_new_tokens = 32 with ignore_eos

Source: run directories subkv-p9f / subkv-p9y: results.jsonl, run log, prefill/decode logs, plan (internal) · 71ef9c61 · 6edc57f4

What changes when the server is already busy?

A successful request on an otherwise idle server gives us a baseline. Next, send it while shorter requests are active.

Does it enter immediately, wait, or get rejected? What happens to the requests already running?

Other checks help establish whether the result is repeatable. Cancel a long request and inspect whether its allocations are reclaimed as expected. Repeat after a restart. Compare cold and warm prefix-cache conditions, then run long enough to observe behavior under sustained load.

Figure 5 · illustration, not data
What must a long-context test record after the request arrives?SCHEMATIC · possible request states, not measured outcomes or durationsSubmittedAdmittedFirst tokenMay wait or be rejectedRecord queue / reasonMay complete or failRecord output / reasonCancellationCheck allocation releaseRepeat with short requests active; measure their waiting time as well.Admission, first token and completion are separate evidence.
What must a long-context test record after the request arrives? Illustration only. Admission does not establish completion or answer quality. A request may also fail or be cancelled before its first token; this simplified map is not an exhaustive state machine. Under background load, record impact on active short requests and allocation reclamation after cancellation.
scope
Conceptual illustration only; no measured memory amounts, durations or rates
scale
Not to scale; example stages are not the tested topology

Source: Supplied F2 draft: What changes when the server is already busy?

These tests turn a single successful run into a description of the conditions the service can support.

What should we mean when we say the deployment supports 1M context?

We should be able to name the configuration, explain the input and output budgets, and describe the workload under which the request was admitted and completed.

Latency and task quality need their own results. A request may fit but take too long for its intended use. It may return an answer without successfully using the information in the long context.

A useful deployment claim makes those conditions visible: how much context was processed, what else the server was doing, how long the response took, and whether it passed the task-level check.

That gives someone running a similar workload enough information to judge whether the result applies to them.

Method

ModelGLM-5.3, 2-bit GGUF (IQ2_XXS routed experts)
Hardware / layoutPrefill/decode disaggregated on RTX 5090 hosts; P PP8, D PP4
RuntimeSGLang fork; 1M runs on builds 71ef9c61 and 6edc57f4; default package b47c730d
Context settingP pool 49,344 × ratio: 10 → 493,440 tokens (default package); 22 → 1,085,568 (245 s run); 46 → 2,269,824 (202 s run)
WorkloadOne long request; the 202 s run had 32 concurrent background requests
Background loadnot recorded
Metricadmitted · completed · TTFT; exact input token count and finish reason not recorded
Repeatsnot recorded

Evidence and references

Code and data for this note will be released as open source; more is coming.

What this does not show

An old TTFT cannot be carried over to another configuration; the complete response and task-quality results for the 1M request are still to be added.