The Hidden Machinery Behind an AI Response

How batching, memory, scheduling, and distributed systems turn shared GPUs into a seamless stream of tokens.Type a question, watch the answer appear word by word, and it’s easy to believe a single mind is on the other end, thinking only about you. It isn’t. What you’re seeing is the surface of a…

How batching, memory, scheduling, and distributed systems turn shared GPUs into a seamless stream of tokens.Type a question, watch the answer appear word by word, and it’s easy to believe a single mind is on the other end, thinking only about you. It isn’t. What you’re seeing is the surface of a fast, messy balancing act, memory limits, hardware physics, and distributed-systems tricks working in milliseconds to keep the effect seamless.This is a tour of that machinery: how a modern inference system handles a flood of wildly different requests on a shared GPU cluster without buckling. By the end, you’ll notice how little of it is actually about AI.The core tension: you are synchronous; the hardware is notEverything follows from one mismatch.You experience the system synchronously. You send one request and wait for one steady stream of text to come back. Simple, linear, personal.The hardware works asynchronously. A GPU isn’t a fast sequential thinker; it’s a wide parallel engine that wants to grind through enormous batches of unrelated numbers in lockstep. At any instant it’s part-way through thousands of requests from thousands of strangers.Bridging those two, turning the hardware’s scattered parallel work into your clean personal stream, is the entire job.A hospital emergency department is a closer picture than it sounds. Patients arrive together with completely different needs: one wants a bandage, one needs hours of surgery, one just has a question. You can’t spin up a private hospital for each, and you can’t know in advance how long any case will take.The whole discipline is treating everyone at once, off shared and scarce resources, without letting anyone quietly deteriorate in the waiting room. That’s the shape of the problem.Why you can’t just batch requestsA GPU is expensive, and feeding it one request at a time wastes almost all of it. The obvious answer is to batch, process many requests together. That’s right for the hardware and wrong for language models, at least in its simplest form.That simplest form is static batching: gather a fixed group, lock them into one block, and hold every answer until the last one is done. Perfect for uniform work like image classification. For text generation, it collapses.Say 50 requests share a batch. Forty-nine ask something small and finish in a handful of tokens. The fiftieth pastes in a report and wants a long essay. Because the group is fixed, the 49 that are done can’t leave, they’re stuck in the batch while the one long request keeps going.It’s the supermarket express lane when the person in front unloads a month’s groceries. Everyone behind them finished shopping ages ago; the belt just won’t move.This is head-of-line blocking, and it’s worse than a queue. GPU matrix math needs the batch’s dimensions to stay consistent on every pass. So the chip keeps running wasted compute cycles on the 49 finished requests, over and over, for thousands of steps — burning tens of thousands of dollars of compute on empty math, just to satisfy the shape of the matrix.Continuous batching: rebuild the batch 40 times a secondThe fix is to redefine the unit of work. Static batching’s unit is a whole request. Continuous batching shrinks it to a single iteration, one pass through the model that yields exactly one new token for each active request.Now requests are bound together only for the instant it takes to produce one token. The moment a pass ends, the scheduler reshuffles: anything that just finished is released and returned, and a waiting request drops into the freed seat. Then it runs again, 20 to 40 times a second.Batching StrategiesWouldn’t interrupting a request that often corrupt it? No, because each pass is stateless. A forward pass is a pure calculation. Producing token 8 doesn’t rely on the act of having produced token 7; it only needs the model’s weights and the text so far.Think of a chess engine handed a board. It works out the next move from the position in front of it, needing no memory of how the pieces got there. Every pass starts fresh from the current state.With no carry-over between passes, the scheduler can pull requests in and out at will, and the results stay exact.The real bottleneck isn’t compute; it’s memoryIf the cores keep no state, where does the conversation live? The answer points at the true constraint in modern AI, and it isn’t the flashy one.It isn’t compute. It’s memory, the fast memory bonded onto the GPU itself, called High Bandwidth Memory (HBM). Ordinary server RAM can’t feed tens of thousands of cores fast enough; they’d stall waiting. HBM sits right against the processor and moves terabytes a second, but it’s costly to make, so there’s precious little of it.Here’s the arithmetic. A 70-billion-parameter model, stored at reduced precision, still takes about 70 GB just to hold, and all of it has to sit in HBM before a single token is generated. Even a top data-center GPU like NVIDIA’s H100 has only 80 GB of HBM to begin with.memoryThat leftover 10 GB is where the whole concurrency fight happens.What eats the memory: the KV cacheThe spare memory holds each conversation’s live state: the KV cache. It’s a by-product of attention: to choose the next word, the model weighs how every earlier word relates to the rest, saving a key and value vector for each token. Skip that cache and it would re-derive all those relationships on every single word, stretching one reply into minutes.Those vectors add up fast. A single 4,000-token exchange can occupy around 10 GB, the GPU’s entire free space.Left unmanaged, a $30,000 chip can serve exactly one person at a time. That’s not a business.And it gets worse. Because running out of memory mid-generation crashes the request, a cautious system plays it safe and asks up front: what’s the most this conversation could ever need? If the ceiling is 4,000 tokens, it reserves the full 10 GB block immediately, even if the user only wants a five-word reply, and that whole block is locked away from everyone else.It’s like booking a freight container for a parcel that turns out to be a matchbox. This is known as the pre-allocation disaster.Paged Attention: virtual memory for the GPUThe breakthrough revives an old operating-system idea: paging. Rather than one huge reservation, memory is cut into small fixed blocks, roughly 16 tokens each, issued only as the text actually grows. Write ten tokens, use one block, return it.pagingCrucially, the blocks needn’t be neighbors. A small mapping table records their order, so blocks scattered across memory still read back as one clean sequence. That lifts a GPU from one user to 30–60, the trick that turns serving these models from a money pit into a business.The architecture at a glancePulled back, the system is a loop: a request travels from client to gateway to scheduler to a pool of GPU workers, and the tokens they produce are steered back to the exact connection waiting for them.ArchitectureEvery box hides its own hard problem. We’ve done the scheduler and the workers; now the pieces around them.Two phases in one request: pre-fill and decodeAnswering isn’t one smooth activity. Memory may be solved, but a new timing problem appears: inference runs in two very different phases that lean on the hardware in opposite directions.Pre-fill digests the prompt, reading all the input tokens at once to build the starting context. It’s compute-heavy; it hammers the GPU’s tensor cores to build the initial KV cache.Decode produces the reply one token at a time, since each word waits on the one before. It’s memory-heavy, capped by how quickly the weights (~70 GB) can be pulled from HBM for each token.Then they clash. Thirty requests are decoding along nicely when someone submits a 50-page document. A naive scheduler runs that whole pre-fill in one burst, tying up the math units for hundreds of milliseconds — and all thirty streams stall at once.prefill-decodeThe remedy is chunked pre-fill. Each step runs on a fixed token budget: fit all the active decodes first, then spend whatever budget is left on a slice of the new prompt. The big prompt seeps in a few hundred tokens per step, tucked into the spare capacity. Its own first token lands a touch later — but nothing else skips a beat.When a worker dies: fail fastWorkers crash. The question is what to do with a request that was streaming when its worker went down.Ship its state to another worker? No, that live state is far too large to move across the network quickly enough.Quietly rerun the prompt somewhere else? Worse. Models are non-deterministic: the same prompt produces a slightly different answer each time. Graft a fresh, different continuation onto the text the user already read, and the reply contradicts itself mid-paragraph.So the system fails fast: notice the crash, return a clean error, drop the connection, and let the client try again. You give up the pretense of unbroken continuity to keep the system correct and simple. Choosing an honest, visible failure over a clever trick that backfires is exactly what mature engineering looks like at scale.Getting answers back: from anonymous tokens to the right socketInside the cluster, the workers pour out a torrent of tokens that carry no label saying which person each one belongs to. Every user, meanwhile, is sitting on one specific open connection, waiting for their stream. Something has to match that flood of anonymous tokens back to the exact connection waiting for it.That’s the response router’s job. It holds a live map of which token stream belongs to which open connection and pushes each token down the right one — and because the volume is enormous, that map is spread across many machines rather than kept in one place.Underneath all this, the fleet also has to agree on which workers are even alive, the kind of membership and failure-detection state that’s often spread with gossip-style protocols, where each node keeps sharing what it knows with a few others until the whole cluster converges.There’s a quieter decision here too: which worker should take a request. Cache-aware routing sends it to a worker that already holds its context, skipping the costly pre-fill; a lighter version simply keeps a whole conversation on the worker that handled its opening turn. The snag is that the worker with the warmest cache is often the busiest, so routing is a constant tug-of-war between cache reuse and load.How it actually breaksRun this at a hundred thousand requests a second and its failure modes become familiar. They come in two flavors: acute (sudden outages) and chronic (slow rot).failure modesThe acute failures are the ones that page someone at 3 a.m. The sharpest is the out-of-memory cascade: an oversized request pushes a worker to the edge of its memory and kills it, taking down the state of the 30–60 requests batched alongside it. Worse, if the client retries, the same payload lands on the next worker and topples that one too, a single bad request rolling straight through the fleet.The chronic failures are the quiet opposite: nothing crashes, but the system slowly rots. In hot-tenant starvation, no request errors and the dashboards look perfectly healthy, utilization pinned near full, yet one heavy customer is soaking up most of the capacity, and everyone else just feels the whole system drag.Fairness, and why it’s hardFixing that starvation is trickier than it looks. The move is to stop serving strictly in arrival order and instead keep a running tally of what each tenant has recently consumed, then let whoever’s used the least go next, backed by per-tenant limits and throttling that only bites when the system is genuinely full.The friction is that strict fairness pulls against both throughput and cache reuse. Spread one tenant’s work evenly, and you scatter their cache, forcing recomputation; concentrate it on one warm worker, and you risk crowding everyone else out.A real scheduler is forever trading off fairness, cache locality, and utilization, never able to max all three.Where engineering quietly turns into businessGo far enough up the stack, and the toughest problems stop being technical.Consider admission control, the doorman. At peak, GPUs run out, and somebody has to be refused. Pure arrival order is perfectly even-handed and a fast way to lose your best customers, because requests aren’t equal.Turning away a hobbyist is a shrug; turning away an enterprise account on a seven-figure contract with a guaranteed response-time clause, because a free user arrived a heartbeat earlier, is a broken agreement. So the door has to know who is knocking, look up their tier on the spot, and even hold capacity in reserve for premium traffic before the cluster is full.At that point the load balancer has stopped shuffling packets and started enforcing sales contracts.The sharpest version is prefix caching. Plenty of applications prepend the same big block of instructions to every small query; imagine a 50-page policy document followed by a thousand different questions about it. A clever system processes that shared prefix once and lets the other 999 queries reuse it. Elegant, until it hits the invoice:Bill per token, and you charge for all 1,000 copies of that document, though it was processed a single time. The margin is effectively unbounded; you’re billing for work you never did.Bill for real compute, and that customer pays a pittance for huge requests. Worse, to reuse the cache, the router steers all 1,000 queries onto the same worker, recreating hot-tenant starvation on that node while collecting almost nothing. Your best optimization becomes an outage you inflicted on yourself.There’s no tidy resolution. At this scale, running the system is as much pricing and game theory as it is engineering.Beyond a single data centerTwo more ideas stretch it across the globe. Within a region, you can split the phases onto separate machines, pre-fill nodes and decode nodes, so a heavy pre-fill can’t stall anyone’s decoding, at the price of moving the KV cache between them.Across regions, each data center keeps its own copy of the model and serves the users nearest to it, overflowing elsewhere when it fills. And since a request’s cache is tied to one machine, you never haul a live request around the world; you place it once, and if it breaks, you fail fast and let it retry.Key takeawaysUsers are synchronous; the hardware isn’t. You wait on one clean stream of text; the GPU grinds through thousands of unrelated requests in parallel. Reconciling those two worlds is the entire job.Batch continuously, not statically. Rebuilding the batch every token, rather than locking it from start to finish, keeps one long request from holding everyone else hostage.Memory is the real ceiling, not compute. Model weights claim ~70 GB of an 80 GB GPU, so every live conversation fights over the ~10 GB that’s left.PagedAttention is virtual memory for the KV cache. Handing out small blocks on demand, instead of reserving the maximum up front, takes a GPU from serving one user to 30–60.Pre-fill and decode pull in opposite directions. One is compute-bound, the other memory-bound; chunking the pre-fill interleaves them so a giant prompt doesn’t freeze every active stream.Fail fast when a worker dies. Migrating or silently rerunning a half-finished request is worse than a clean error the client can retry.At scale, the hardest problems turn economic. Admission control, fairness, and how you bill a shared prefix cache are business decisions dressed as engineering.The punchline: it was never about intelligenceStep back and notice what never entered the story: intelligence.At this layer, the model for all its billions of parameters is completely hidden behind an interface. To the infrastructure, it’s just a bulky lump of matrix multiplications to be shoveled around.The genuine difficulty of serving AI is an ordinary distributed-systems grind: scheduling work reactively, keeping countless stateful connections alive over flaky networks, squeezing every byte of memory, and steering traffic on the fly. The same bottlenecks engineers wrangled decades ago, now at a scale that makes them bite again.You can build the most powerful engine ever made, but without a transmission, cooling, and fuel lines, it’s an expensive paperweight. The model is the engine. All of this is the drivetrain.Which leaves a question worth sitting with: how much of the AI wave now driving trillion-dollar valuations is, underneath, a rediscovery of unglamorous plumbing — problems we assumed were settled — back to bite us at a scale that makes them hard all over again?This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!The Hidden Machinery Behind an AI Response was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: Generative AI Pub — Published August 12, 2026 at 14:43 UTC — Category: Image AI