What AI Teams Should Know About HBF and Tiered Memory

For years, most of the AI memory conversation has been about bandwidth: can memory feed GPUs fast enough? Frontier inference is adding a second problem. Model weights, KV caches, and expert weights are getting large enough that there may not be enough HBM to keep everything nearby, even though only…

For years, most of the AI memory conversation has been about bandwidth: can memory feed GPUs fast enough? Frontier inference is adding a second problem. Model weights, KV caches, and expert weights are getting large enough that there may not be enough HBM to keep everything nearby, even though only a fraction of that state needs maximum bandwidth at any given moment. I think that gap is what makes High Bandwidth Flash interesting. HBF potentially adds a much larger memory tier alongside HBM, with HBM holding the hot working set and HBF keeping more of the less frequently accessed state close to the accelerator. One caveat up front: HBF is still early, and many of the architectures and results I discuss below are modeled rather than measured on production hardware. Table of Contents More Memory Can Mean Less Data Movement The KV Cache Is Not One Thing Memory Placement Becomes Part of Inference Don’t Buy a GPU to Solve a Memory Problem Read-Heavy Workloads Fit HBF Best Prepare for Tiered Memory Now More Memory Can Mean Less Data Movement Two examples make this concrete for me. Mixture-of-experts models are the cleanest example: a model might carry dozens of specialized experts, but each token typically activates only a handful of them. Keeping more of that expert pool near each accelerator means fewer trips across the network to reach an expert that happens to live on another chip. One modeled setup that raised locally resident experts from 8 to 32 cut that kind of cross-node traffic to roughly a sixth of what it was before. I’d go a step further than that result suggests: this makes memory capacity partly a networking decision, and any team running a large MoE deployment should be planning the two together rather than budgeting for them separately. The same pattern shows up when you’re serving many models instead of one big one. Teams serving multiple frontier models, fine-tuned models, or customer-specific variants already pay a real tax when everything cannot stay in memory. Every model swap adds overhead, and teams can measure that cost today. In one modeled case covering 27 models, simply having enough room to keep all of them resident dropped the typical time to first token from around 196 milliseconds to under 7. The exact number isn’t really the point. What matters is that the improvement came largely from avoiding data movement, not from making the computation itself faster. Return to TOC The KV Cache Is Not One Thing It’s tempting to talk about “the KV cache” as if it’s one uniform pile of data, and I think that’s the wrong way to look at it. KV cache is the temporary state a model keeps from earlier tokens so it can reuse that work as it generates the next one. Recent KV, older KV, shared prefixes, and the private cache that grows during a single request can have very different access patterns. How hot the data is, how often it gets reused, and how often new data gets written can matter as much as the cache’s overall size when you’re deciding where any of it should live. There’s a cautionary result here worth taking seriously. In one tested system, simply swapping faster flash-based memory in for the existing overflow storage didn’t speed anything up. It made the system slower. The reason traces back to exactly the distinction above: the data that ended up in that tier received more writes than reads, while much of the frequently reused material had already stayed in faster memory. Flash is a poor fit for sustained write-heavy workloads. A separate analysis found the reverse can also be true: KV access in sufficiently long contexts can become quite read-heavy, since KV is created once and may be referenced many times afterward. Put those two findings together and the practical lesson is simple: context length alone won’t tell you where KV cache belongs. The actual read, write, and reuse pattern will, so measure that before you decide. Return to TOC Memory Placement Becomes Part of Inference This puts more responsibility on the inference runtime, the software layer that orchestrates how a model is served. In a tiered system, serving software doesn’t just decide where computation runs. It also has to decide where model state should live, what stays in HBM, what can sit in HBF, and what needs to move next. Those choices can change with batch size, context length, request mix, or which experts are being used most often. One of the key tricks is prefetching: move upcoming data into HBM while the GPU is still working on what it already has. Get that right and much of the slower tier’s latency can be hidden. Get it wrong and an expensive GPU ends up waiting for data. That’s why peak bandwidth alone won’t tell you how well a tiered memory system will perform. I would also start paying more attention to memory locality in inference telemetry. Alongside GPU utilization, latency, and throughput, teams should know how much state is genuinely hot, how often models or experts are fetched remotely, how much KV gets reused, and how much time is lost waiting on memory. Those measurements can help distinguish a compute problem from a capacity, caching, or placement problem. That distinction matters more as the memory hierarchy gets more complicated. Return to TOC Don’t Buy a GPU to Solve a Memory Problem One practical implication is easy to miss: sometimes a team needs another GPU because it needs more HBM, not more compute. If eight GPUs are required because the workload genuinely needs eight GPUs’ worth of computation, more memory will not help. But if eight are required because the model will not fit in four GPUs’ HBM, the economics look very different. More local capacity could potentially reduce not just GPU count, but cross-GPU traffic, networking, power, and the overhead of moving models around. That’s why I wouldn’t evaluate HBF on dollars per gigabyte alone. A cheap memory tier can get expensive quickly if it leaves an expensive GPU waiting for data. A slower tier can be a good trade if it keeps more useful state nearby and lets the system use fewer accelerators. I think the better question is what the entire system costs to produce a useful token at the latency and throughput you need. Return to TOC Read-Heavy Workloads Fit HBF Best If you’re wondering where to place your bets on HBF over the next year or two, I’d point you squarely at inference. The clearest candidates, model weights, MoE expert weights, and the multi-model serving case I mentioned earlier, share one useful property: they are read repeatedly but rewritten rarely. Training has almost the opposite profile. Weights, gradients, and optimizer state change constantly, which is a much tougher workload for NAND flash, the technology used in SSDs, because writes are slower and the cells have finite endurance. So for the next 6 to 18 months, I’d track HBF as part of how frontier inference architecture is evolving, not as a new foundation for training. The write side deserves more scrutiny than the headline bandwidth numbers usually get. One HBF study modeled workloads whose write rates exceeded the budget assumed for a reasonable device lifetime. In the same analysis, sustained writes generated enough heat to reduce achievable bandwidth well below the interface peak. That’s a useful lesson beyond HBF. When evaluating a new memory tier, I would ask for sustained performance and endurance under your actual access pattern, not just peak read bandwidth. The spec sheet tells you what the interface can do. Your workload determines what the system can sustain. Return to TOC Prepare for Tiered Memory Now The architecture is still moving, so I wouldn’t build a 6 to 18 month roadmap around HBF’s exact shipping date. What I would bet on is the pattern underneath it: frontier inference is heading toward hot state that belongs in HBM, warm state that needs to stay close to the accelerator without requiring HBM’s full speed, and cold state that can live farther away. Whether HBF fills that middle tier, or it ends up being larger HBM, GDDR, CXL-attached memory, or some combination, the software problem looks similar. That’s the broader point I’d focus on. You don’t need HBF hardware to start benefiting from thinking harder about model residency, KV reuse, expert locality, prefetching, caching, and memory-aware scheduling. Those problems already exist in frontier inference. HBF didn’t create them, but it is making them much harder to ignore. Teams that get better now at understanding where state lives and how it moves will be better prepared for whichever memory architecture ends up winning. Return to TOC Subscribe to our weekly newsletter Sources Hot Chips 2026: Applying High Bandwidth Flash (HBF) Potential Applications of HBF in LLM Serving Systems HBF Sucks! A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving HBF Technolgy, Workload Analysis and Roadmap Sandisk Unveils The Future Of Memory Architecture For Ai Introducing: High Bandwidth Flash Return to TOC The post What AI Teams Should Know About HBF and Tiered Memory appeared first on Gradient Flow.

Source: Gradient Flow — Published August 25, 2026 at 11:00 UTC — Category: Models