Your LLM Isn’t Thinking — It’s an Engineer Pulling Weights at 10,000 Tokens Per Second
What really happens inside an AI model, and why the engine running it matters more than you think.When most people interact with ChatGPT, Claude, or any large language model, they imagine something magical — a digital brain, thinking, reasoning, forming sentences the way humans do.What’s...
What really happens inside an AI model, and why the engine running it matters more than you think.When most people interact with ChatGPT, Claude, or any large language model, they imagine something magical — a digital brain, thinking, reasoning, forming sentences the way humans do.What’s actually happening is far more interesting, and far more mechanical.Your LLM is not thinking. It is an engineer — a very fast, very precise engineer — pulling weights off a shelf, multiplying them together millions of times per second, and handing you back a probability distribution dressed up as a sentence.And the thing that puts that engineer to work? That’s called an inference engine.This is the story of what inference engines are, why they matter enormously, and why one of them — vLLM — is quietly changing how the world serves AI at scale.Part 1: What an LLM Actually Does (The Weight-Pulling Analogy)Imagine a warehouse. An enormous warehouse — so large it takes 140 gigabytes just to store its inventory. Every shelf holds a number. Billions of numbers. These are the weights — the learned parameters of the model, the result of months of training on trillions of words.Now imagine an engineer who works in this warehouse. Every time you send a message, the engineer gets to work:Reads your message — tokenises it into chunks (words, subwords, characters)Walks to the right shelves — fetches the weights relevant to your inputMultiplies, adds, transforms — runs your tokens through layer after layer of matrix operationsPicks the next word — samples from a probability distribution over the entire vocabularyRepeats — walks back to the shelves, fetches more weights, picks the next wordKeeps going — until the response is completeThat’s it. That is what your “thinking” AI is doing. There is no reasoning in the human sense. There is no understanding. There is an engineer, doing maths, very very fast, one token at a time.Your prompt: "What is the capital of France?"Token 1 generated: "The" ← engineer picks from vocabularyToken 2 generated: "capital" ← engineer picks againToken 3 generated: "of" ← and againToken 4 generated: "France" ← and againToken 5 generated: "is" ← still goingToken 6 generated: "Paris" ← finally, the answerToken 7 generated: "." ← doneEach token generation is a full trip through the warehouse. Fetch weights. Multiply. Output one token. Repeat.This is why LLMs are slow when serving many users at once — each user’s engineer is making trips through the same enormous warehouse, and the warehouse only has so many doors (GPU memory bandwidth).The inference engine is the warehouse management system — it decides how to organise the shelves, how many engineers can work simultaneously, and how to stop them getting in each other’s way.Part 2: What Is an Inference Engine?In classical AI, an inference engine was the reasoning component of an expert system — it applied logical rules to a knowledge base to derive conclusions. Think of it like a judge: given a set of facts and laws, it reaches a verdict.In the modern LLM world, the definition has evolved. An LLM inference engine is the software layer that:Loads the model weights onto GPU memoryAccepts incoming requests (your prompts)Runs the model’s forward pass — the mathematical trip through the warehouseReturns generated tokens efficientlyManages GPU memory, batching, and scheduling across many simultaneous usersThink of the difference between the model and the inference engine this way:The model is the recipe. The inference engine is the industrial kitchen that cooks it for ten thousand customers simultaneously without burning anything.Without a good inference engine, even the most powerful model crawls. With a great inference engine, a smaller model can outperform a larger one in production simply because it serves faster, cheaper, and at higher throughput.Part 3: The Inference Engine LandscapeBefore we go deep on vLLM, here’s the lay of the land. Several inference engines exist today, each with different strengths:🔵 TensorRT-LLM (NVIDIA)Built by NVIDIA specifically for their GPU hardware. Think of this as a Formula 1 car — the fastest possible option, but only on NVIDIA tracks. It uses highly optimised CUDA kernels, quantisation, and graph compilation to extract maximum performance from H100 and A100 GPUs.Best for: Production deployments on NVIDIA hardware where raw speed is the top priority. Tradeoff: Complex setup, NVIDIA-only, steep learning curve.🟢 OllamaThe friendly neighbourhood inference engine. Designed to run LLMs on your laptop, Mac, or local machine with a single command. Wraps llama.cpp under the hood and provides a clean API.ollama run llama3# That's it. LLM running locally.Best for: Local development, experimentation, privacy-sensitive use cases. Tradeoff: Not designed for high-throughput production serving.🟡 llama.cppThe legendary C++ library that brought LLM inference to CPUs and consumer hardware. Supports quantisation formats like GGUF that compress a 70B model down to something that can run on a MacBook Pro.Best for: Resource-constrained environments, edge devices, CPU-only machines. Tradeoff: Slower than GPU-based engines for large-scale serving.🟠 TGI — Text Generation Inference (Hugging Face)Hugging Face’s production serving framework. Deep integration with the HF ecosystem, supports a wide range of models, and has a clean OpenAI-compatible API. A solid, battle-tested choice with good documentation.Best for: Teams already in the Hugging Face ecosystem. Tradeoff: Not as aggressively optimised for throughput as vLLM.🔴 SGLang (Stanford)A newer engine focused on structured generation — situations where you need the model to output valid JSON, follow a grammar, or complete complex multi-step programs. Introduces a programming language for LLM interactions that enables advanced batching optimisations.Best for: Agentic workflows, structured outputs, complex multi-call pipelines. Tradeoff: Newer, smaller community, still maturing.🟣 DeepSpeed-MII (Microsoft)Microsoft’s inference framework built on DeepSpeed. Strong support for very large models and multi-GPU deployments, with good integration into the Azure ecosystem.Best for: Microsoft/Azure environments, very large model serving. Tradeoff: More complex setup, tightly coupled to the DeepSpeed ecosystem.Part 4: vLLM — The Engineer Who Rewrote the WarehouseIn 2023, a team at UC Berkeley published a paper that changed how everyone thought about LLM serving. The paper introduced PagedAttention, and the system built around it was called vLLM — the “v” standing for “virtual”, a direct nod to virtual memory in operating systems.vLLM is not just faster. It is architecturally smarter — solving problems that previous engines either ignored or patched around. Let’s go through each innovation.Innovation 1: PagedAttention — The Virtual Memory of AIFirst, what is the KV Cache?When your engineer walks through the warehouse generating tokens, they need to remember what they’ve already computed. Specifically, for every token already generated, the model computes two vectors — a Key and a Value — that are needed when computing attention for future tokens.Attention formula:Output = softmax(Q · Kᵀ / √d) · VQ = Query (what the current token is looking for)K = Key (what each past token offers)V = Value (what each past token contains)Rather than recomputing K and V for all past tokens at every step (which would be catastrophically slow), they are cached in GPU memory — this is the KV cache.The KV cache is the single biggest consumer of GPU memory during inference. And before PagedAttention, it was managed terribly.The old approach — and why it was wastefulOld systems pre-allocated a large, contiguous block of GPU memory for each request, sized for the maximum possible sequence length:Max sequence length = 2048 tokensActual response length = 312 tokensReserved memory: [■■■■■■■■■■■■■■■■■■■■□□□□□□□□□□□□□□□□□□□□] used (312) wasted (1,736 tokens worth of memory)This meant:60–80% of GPU memory was sitting empty, reserved but unusedFewer requests could run simultaneouslyGPUs were underutilised despite having free memoryPagedAttention — the OS solutionThe insight from the Berkeley team was elegant: operating systems solved this exact problem 50 years ago with virtual memory and paging.In an OS:Program sees: [Page 0][Page 1][Page 2][Page 3] ← logical, contiguousPhysical RAM: [Frame 7][Frame 2][Frame 11][Frame 4] ← scattered, non-contiguousPage Table: maps logical pages → physical framesPagedAttention does the identical thing for KV cache:Model sees: [KV Block 0][KV Block 1][KV Block 2] ← logicalGPU Memory: [Slot 9] [Slot 3] [Slot 14] ← physical, scatteredBlock Table: maps logical KV blocks → physical GPU memory slotsFixed-size blocks (e.g., 16 tokens per block) are allocated on demand as tokens are generated. The last block may be partially filled, but waste is bounded to at most block_size - 1 tokens — compared to hundreds or thousands of wasted tokens in the old approach.Request A generating token 35 (block size = 16):Block 0: tokens 1-16 → Physical Slot 4 [■■■■■■■■■■■■■■■■] fully usedBlock 1: tokens 17-32 → Physical Slot 9 [■■■■■■■■■■■■■■■■] fully usedBlock 2: tokens 33-35 → Physical Slot 1 [■■■□□□□□□□□□□□□□] 3/16 used (last block)Maximum waste: 15 tokens. That's it.The bonus — Copy-on-Write for beam searchWhen exploring multiple response paths (beam search), old engines had to duplicate the entire KV cache for each beam — a massive memory cost.With PagedAttention, beams share KV blocks with a reference counter, copying only when a beam actually diverges — exactly like the Unix fork() system call:Parent sequence: [Block 0][Block 1][Block 2] ↑ ↑ ↑Beam 1 (shares): [same] [same] [copy only here, when beams diverge]Beam 2 (shares): [same] [same] [copy only here, when beams diverge]The result: vLLM can fit 2–4× more concurrent requests on the same GPU compared to previous systems — purely from memory management.Innovation 2: Continuous Batching — Never Let the GPU IdleGPUs are parallel machines. Running one request on a GPU is like hiring a 10,000-person factory to make a single sandwich. You need to keep all the workers busy.Static batching — the old, wasteful wayBatch starts → [Req A, Req B, Req C, Req D] all start together Req B finishes early (short response) Req B's GPU slots sit IDLE waiting for A, C, D Only when ALL finish → next batch startsShort requests waste GPU cycles. New requests queue up even when the GPU has spare capacity. The analogy: a restaurant kitchen that cooks 4 dishes, serves all 4 at the same time to the same table, then takes the next order — even if some dishes were ready 10 minutes earlier.Continuous batching — iteration-level schedulingvLLM makes a scheduling decision after every single token generation step:Iteration 1: GPU runs [Req A, Req B, Req C, Req D]Iteration 2: GPU runs [Req A, Req B, Req C, Req D]Iteration 6: Req B finishes → slot freedIteration 7: GPU runs [Req A, Req E, Req C, Req D] ← Req E jumps in immediatelyIteration 11: Req D finishes → slot freedIteration 12: GPU runs [Req A, Req E, Req C, Req F] ← Req F jumps in immediatelyThe GPU is never idle. Every iteration, every slot is filled. New requests don’t wait for an entire batch to complete — they slide in the moment a slot opens.The analogy updated: the kitchen now finishes one dish, sends it out immediately, and starts the next order right away — rather than holding everything until the whole table’s food is ready.Real-world impact: vLLM demonstrated up to 24× higher throughput than a naive Hugging Face pipeline serving setup — largely due to continuous batching combined with PagedAttention.Innovation 3: Chunked Prefill — Fair Time for EveryoneThe problem — long prompts are bulliesEvery LLM request has two phases:Prefill: Process the entire input prompt in one go (parallel, fast). Decode: Generate output tokens one at a time (sequential, slower).When a user sends a 4,000-token prompt (a long document for review, say), the prefill phase monopolises the GPU:Without chunked prefill:GPU processing Request A's 4000-token prefill: ████████████████████ (400ms)All other requests: ░░░░░░░░░░░░░░░░░░░░ STALLED for 400msEvery other user experiences a 400ms freeze in their response.Chunked prefill — slice the long promptvLLM breaks the long prompt into chunks (e.g., 512 tokens each) and interleaves them with ongoing decode steps:Iteration 1: [Req A chunk 1/8] + [Req B token gen] + [Req C token gen]Iteration 2: [Req A chunk 2/8] + [Req B token gen] + [Req C token gen]Iteration 3: [Req A chunk 3/8] + [Req B token gen] + [Req C token gen]...Iteration 8: [Req A chunk 8/8] + [Req B token gen] + [Req C token gen] → Req A's first token generatedReq A still gets processed fully, but it no longer blocks everyone else. Time-to-first-token for concurrent users drops dramatically.Innovation 4: Speculative Decoding — Draft Fast, Verify SmartThe fundamental bottleneckHere is a counterintuitive truth about GPU inference: generating 1 token and generating 5 tokens costs nearly the same time.Why? Because the bottleneck is not computation — it’s memory bandwidth. The GPU spends most of its time loading the model’s 140GB of weights from High Bandwidth Memory (HBM) into compute units. That loading cost is the same whether you use the weights to produce 1 token or 5.Speculative decoding exploits this.The draft-and-verify trickStep 1 — Draft (small model, fast): 7B model generates 5 candidate tokens in ~5ms: ["The", "COI", "declaration", "appears", "valid"]Step 2 — Verify (large model, one forward pass): 70B model checks all 5 in parallel: ["The" ✓, "COI" ✓, "declaration" ✓, "appears" ✗] accepted rejectedStep 3 — Accept + correct: Accepted: "The COI declaration" Large model provides correct token for "appears" position Net result: 3 tokens produced for ~1 large model forward pass costIf the small and large model agree 80% of the time (which they do when from the same model family), you get 3–4 tokens accepted per round — a 3–4× effective speedup with zero loss in output quality, because the large model still validates everything.Innovation 5: Prefix Caching — Don’t Recompute What You’ve Already DoneIn almost every production LLM application, requests share a common prefix — a system prompt, a set of instructions, a document being analysed:Request 1: [System prompt: 800 tokens][User query A: 50 tokens]Request 2: [System prompt: 800 tokens][User query B: 30 tokens]Request 3: [System prompt: 800 tokens][User query C: 70 tokens]Without prefix caching, the KV cache for those 800 system prompt tokens is recomputed from scratch for every single request. Wasteful.vLLM hashes each block of tokens:Block content → SHA256 hash → cache keyRequest 1: compute KV for system prompt blocks → store under hash H1Request 2: compute hash of system prompt → H1 found in cache → REUSERequest 3: compute hash of system prompt → H1 found in cache → REUSEOnly the unique user query tokens need fresh computation.Real-world saving: If your system prompt is 1,000 tokens and your user query is 200 tokens, prefix caching reduces computation per request by 83% for the prompt portion.For applications like document review, legal analysis, or compliance checking — where the same policy documents are referenced repeatedly — this is transformative.Innovation 6: Tensor & Pipeline Parallelism — Scaling Across GPUsA 70B parameter model in FP16 requires 140GB of GPU memory. A single A100 GPU has 80GB. The model doesn’t fit.Tensor Parallelism — split the mathsEach weight matrix is split across multiple GPUs:Weight matrix W: GPU 1 handles: W[:, 0:4096] ← left half of columns GPU 2 handles: W[:, 4096:8192] ← right half of columnsInput X broadcast to both GPUs simultaneously: GPU 1 computes: X · W[:, 0:4096] → partial result GPU 2 computes: X · W[:, 4096:8192] → partial resultAllReduce combines results → full outputWall-clock time: same as 1 GPU, but with 2× the memoryScaling:1 GPU (80GB) → serves up to ~40B model2 GPUs (160GB) → serves up to ~80B model 4 GPUs (320GB) → serves up to ~160B model8 GPUs (640GB) → serves Llama 405B comfortablyPipeline Parallelism — split the layersTransformer layers are divided across GPUs like an assembly line:GPU 1: Layers 1-20 → processes input, passes to GPU 2GPU 2: Layers 21-40 → processes, passes to GPU 3GPU 3: Layers 41-60 → processes, outputs tokenDifferent batches flow through the pipeline simultaneously:Batch 1: GPU3 ←── GPU2 ←── GPU1 (still processing batch 2)Batch 2: GPU3 ←── GPU2 ←── GPU1 (still processing batch 3)vLLM supports both parallelism strategies simultaneously, allowing teams to serve the largest models (405B+) in production without compromise.Innovation 7: Quantisation Support — More Model, Less MemoryFull-precision models (FP16) are large and slow. vLLM supports multiple quantisation formats natively:LLaMA 70B memory requirements:FP16 (no quantisation): 140GB → needs 2× A100 80GBINT8 (8-bit): 70GB → fits on 1× A100 80GBINT4 (4-bit, GPTQ/AWQ): 35GB → fits on 1× A100 40GBFP8 (H100 native): 70GB → hardware-accelerated on H100Critically, modern quantisation techniques (especially AWQ) lose less than 1% accuracy compared to full precision — while halving or quartering memory requirements.For production teams, this means serving a 70B model on hardware budgeted for a 35B model.Part 5: How It All Comes TogetherEvery vLLM innovation targets a specific bottleneck. Together, they form a coherent system:The inference engine is arguably more important than the model for most production AI applications.Consider: a team running GPT-4 equivalent quality with an open model on vLLM can:Serve 10× more users on the same hardwareReduce cost per token by 80% compared to naive servingKeep data on-premise for compliance and privacyControl and customise their serving stack entirelyThe model is the recipe. But the inference engine determines whether you can feed ten people or ten thousand.vLLM — with PagedAttention, continuous batching, speculative decoding, prefix caching, chunked prefill, and multi-GPU parallelism — is currently the most complete answer to that challenge in open source.The name says it all: virtual LLM — the same insight that made virtual memory revolutionary for computers is now making AI inference revolutionary for GPU clusters.Your LLM is still an engineer pulling weights. vLLM just makes sure that engineer never stops, never waits, and never wastes a single shelf.If you found this useful, follow for more deep dives into AI infrastructure, LLM serving, and the engineering behind modern AI systems.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!Your LLM Isn’t Thinking — It’s an Engineer Pulling Weights at 10,000 Tokens Per Second was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI