Deep Dive: How Speculative Decoding Makes LLMs Faster
Speculative decoding can make a coding assistant respond faster by having a small model propose several future tokens for a larger model to check together. The larger model, called the target, still determines the output probabilities. The surprising part is that the small model can guess wrong…
Speculative decoding can make a coding assistant respond faster by having a small model propose several future tokens for a larger model to check together. The larger model, called the target, still determines the output probabilities. The surprising part is that the small model can guess wrong without changing those probabilities.In the two-model version we will follow, the target still does its full computation. The saving comes from arranging that computation differently, so one pass through its layers can advance the answer by several tokens. Whether that saves time depends on how much of the draft survives.A draft makes several conditional predictions available for one full target pass. Read moreSource: The AI Edge — Published — Category: Research