CUDA Graphs can speed up repeated GPU work by preparing its execution plan once and replaying it with new data. They are a feature of NVIDIA’s CUDA platform. In LLM inference, that work can be the sequence of operations used to generate the next token. Replay launches those operations again; it…
Annons
Annons
CUDA Graphs can speed up repeated GPU work by preparing its execution plan once and replaying it with new data. They are a feature of NVIDIA’s CUDA platform. In LLM inference, that work can be the sequence of operations used to generate the next token. Replay launches those operations again; it does not return an earlier answer or avoid computing the new token.Serving engines such as vLLM expose full and piecewise graph modes, making replay coverage a practical choice about latency and GPU memory.How can a reusable plan handle an LLM whose tokens, context lengths, and active requests keep changing?The useful distinction is between the description of the work and the data that work reads. A captured operation can keep the same input address while the values stored at that address change. The GPU then follows the prepared dependencies and computes a fresh result.Replay reuses the operation description. It still computes from the values present when it runs.A fixed execution plan does not require a fixed answer. An operation that doubles the numbers in an input buffer produces a different output after those numbers change, even though the operation and its input address are unchanged. Replacing the buffer itself is a different matter: the prepared operation may still point to the original storage.LLM serving builds on that distinction. New tokens, positions, and cache locations can become updated data inside persistent buffers. Changes to the amount or arrangement of work may instead require another prepared graph, a smaller captured region, or ordinary execution.We will follow that boundary from GPU launches to growing token histories, then examine when the cost of keeping several graph variants ready outweighs the launch savings.Table of contentsWhere a decoding step spends timeCapture a plan, replay fresh workKeep addresses stable while values changeFit changing LLM state into a replayChoose the right graph for the batchDecide whether replay pays Read more
