Your GPU finished the kernel. What is the next stage waiting for?

You’re profiling a multi-GPU inference run. One stage finishes its computation, but the next stage doesn’t start immediately. There’s a gap in the timeline.

What happens inside that gap?

The activations need to reach the next GPU. Before that can happen, processes may need to coordinate, buffers need to be ready, and execution dependencies need to be satisfied. Each of those steps can contribute to the time a request spends waiting.

If we want to shorten that wait, we first need to understand what it contains.

Are we waiting for the data to move, or for the transfer to begin?

It’s tempting to describe the whole gap as “communication overhead.” That label is useful until we need to decide what to change.

Consider two possible cases. In one, the transfer begins promptly, and moving the tensor takes most of the time. In the other, the tensor is ready, but coordination delays the start of the transfer.

Both leave the next stage waiting. They point to different optimizations.

A faster transfer path could help the first case. In the second, much of the delay may remain even if the copy itself becomes faster. We need to inspect how the handoff is initiated and what must happen before it can proceed.

How do we see what’s happening between the two stages?

Start by following a single handoff through the timeline:

• When does the producing stage finish?
• When is the transfer submitted, and when does data movement actually begin?
• When is the data available to the receiving stage?
• When does that stage begin its computation?

Those events help separate coordination, data movement, and the wait before the receiving computation starts.

They also keep the analysis honest. Work on different CPU threads and GPU streams can overlap, so adding up individual operation durations may not explain the elapsed time. The useful question is which dependencies are holding up this request.

What led us to investigate this in pprelay?

Our work involved a single eight-GPU host whose tested configuration had no usable GPU peer-to-peer transfer path. That made the host-mediated handoff between stages worth examining.

The question was concrete: once a stage had produced its output, what work remained before the next stage could use it?

pprelay explores that handoff using shared-memory coordination and pinned tensor slots between processes on the same machine. It works within the available transfer path; it does not enable hardware P2P.

The aim is to reduce the overhead of passing work between stages. Whether that helps a particular deployment depends on how much of its request latency comes from that overhead.

The separate historical prototype microbenchmark below shows where a host-mediated mailbox beat NCCL and where it did not; it is not evidence of a serving speedup.

If the handoff gets faster, will the request get faster too?

That is the next measurement.

A handoff benchmark can show that a local operation improved. We then need to put it back into the pipeline and check whether the request finishes sooner, or whether the serving system completes more work under the same conditions.

Keep the model, GPU layout, and request distribution fixed. Verify execution ordering and safe buffer reuse alongside performance. Then inspect the timeline again: did the gap shrink, and did that change reach the end of the request?

That is the result we want to understand.

The next time a GPU finishes its computation and the following stage stays idle, zoom in on the space between them. Finding out what it is waiting for gives us a concrete place to start.