What’s new in Kimi K3, a 2.8T parameter model.
Understanding Kimi K3’s architecture and what actually changed from K2.Source: Image by the author, reproduced from the Kimi K3 launch post.Moonshot shipped Kimi K3 on 16 July. 2.8 trillion parameters, a one million token context, native vision. Artificial Analysis scored it 57 on their…
Understanding Kimi K3’s architecture and what actually changed from K2.Source: Image by the author, reproduced from the Kimi K3 launch post.Moonshot shipped Kimi K3 on 16 July. 2.8 trillion parameters, a one million token context, native vision. Artificial Analysis scored it 57 on their intelligence index, about level with Opus 4.8 and GPT-5.5, behind Fable 5 and GPT-5.6 Sol. It went straight to first place on Arena’s frontend code board.Most of the coverage led with Kimi Delta Attention. That is what the blog leads with too. But KDA is not new. It came out in the Kimi Linear report in October 2025, with open kernels and a 48B checkpoint you could download.What is true is that K3 is the first Kimi flagship to use it. K2.5, K2.6, and K2.7 all ran plain MLA. So the news isn’t the mechanism; it’s that the mechanism finally survived a run this size.One of the engineers said the design started in January 2025, which makes it about eighteen months from idea to frontier scale.So if KDA is old, what did they actually change? And why did the model end up at 2.8 trillion parameters?What is new, and what isn’tHere is the K2 to K3 diff as best I can reconstruct it.Source: Image by the author.Detailed Architecure of Kimi K3. Source: Moonshot AI, © Moonshot AI.The KDA row is the one everyone quoted, and it is the one that changed least. Kimi Linear already did 3:1 KDA to MLA. K3 gates the MLA layers and scales the whole thing up, and that is roughly it on that front.The KDA row is the one everyone quoted, and it is the one that changed least. Kimi Linear already did 3:1 KDA to MLA. K3 gates the MLA layers and scales the whole thing up, and that is roughly it on that front.Every other row is new. The one I would put money on mattering most is the residual row, because it has a paper behind it and the paper is good.KDA in one pageYou still need to know what KDA does, so here it is quickly.Keep a matrix S as memory. Plain linear attention just writes into it:https://medium.com/media/f2abfcb1470faeaf523e349c1af95589/hrefNothing ever gets removed. The memory fills up, and everything blurs together. This is why plain linear attention has always been bad.DeltaNet fixes it by asking the memory to do a job: given keyk_t, return value v_t. Take one gradient step on that error, and you get the delta rule:https://medium.com/media/c2b03979ef8593e826c87ef5a79cc72a/hrefErase, then write. The (I - β_t k_t k_t^T) part wipes whatever the memory currently holds for that key before writing the new value. Gated DeltaNet adds a forget gate α_t in front, so old memories fade.KDA changes one thing. The forget gate goes from a single number per head to a vector of 128 numbers, one per channel:https://medium.com/media/ff3a6acd7334d85b96693d07fb6a3716/hrefEvery feature dimension now forgets at its own rate.Source: Image by the author.The bars show how much of each memory channel survives one step. The blocks show what happens when a new value gets written. KDA is the only one that does both jobs well.Two things are worth knowing about it.First, the win is memory, not compute. Work through the cost formula in the paper and at head dimension 128 with chunk size 64, KDA costs about 127,000 FLOPs per token no matter how long the sequence is. Full attention costs 256 times the sequence length. Below roughly 500 tokens, full attention is actually cheaper.KDA wins because it kills the KV cache, which frees the memory that was capping your batch size. At 1M context with batch size 1, Kimi Linear decodes about 2.3x faster than MLA. Spend the freed memory on a bigger batch, and you get 6.3x. That gap is the whole point.Second, the per-channel gate is doing a job you might not expect. Write RoPE out in full, and it is a product of rotation matrices sitting between the query and the key, with a different frequency per pair of dimensions.Write the gated delta rule out, and it has the same shape, except the matrices are learned instead of fixed. So the gate is a positional encoding. A single number per head can’t give you RoPE’s per-dimension frequency spread. 128 of them can.That is why Kimi Linear could drop position encoding from its MLA layers entirely and let KDA carry it. Whether K3 still does that, the blog doesn’t say.That’s KDA. Now the new stuff.Attention ResidualsThis paper, also from the Kimi team, came out in March 2026, and it is the biggest new idea in K3.Start with the problem. The residual connection is h_l = h_{l-1} + f(h_{l-1}). Unroll it, and every layer gets the same evenly weighted sum of everything before it. Layer 60 cannot ask for layer 3.It can only read the running total that has been piling up in between. And with PreNorm, that total grows with depth, so each layer's share of it shrinks.Deeper layers have to shout louder just to be heard, which makes training less stable. It is also why you can delete a chunk of a trained model's layers and barely notice.Look at that again, and the residual stream is an RNN. It squashes everything into one state, and the early stuff gets buried. We already know the fix for that, because the transformer did it to sequences years ago. Use attention.So AttnRes replaces the sum with a softmax over layers:https://medium.com/media/8a67cb8c1e86e91686c56c11e41b3872/hrefv_i is the output of layer i. The weights come fromsoftmax(w_l^T RMSNorm(k_i)), where w_l is one learned vector per layer. That's the whole mechanism. One vector and one RMSNorm per layer.Source: Image by the author.Each dot is one layer working on one token. The blue fan is attention. In K3, it comes off the token axis and goes onto the layer axis, because there are only about a hundred layers and there can be a million tokens.Two details make it work.w_l is a plain parameter, not something computed from the hidden state. That sounds minor. It means you can compute the attention weights for a whole group of layers in one batched matmul, before those layers have run. That is what keeps it cheap.And the vectors must start at zero. Then the softmax is uniform at step one, so AttnRes begins life as exactly the equal-weight sum it replaces. Start them anywhere else and training goes unstable. They checked.Full AttnRes needs every layer’s output kept alive and shipped between pipeline stages, which is O(Ld) memory and traffic. So they group layers into N blocks, sum inside a block, and attend only over the N block summaries. That drops to O(Nd).About 8 blocks get most of the benefit. Set N=1 and you are back to a normal residual. Set N=L, and you have the full version.The numbers: Block AttnRes matches a baseline trained with 1.25x more compute. Training overhead under pipeline parallelism is under 4%. Inference overhead is under 2%.On the 48B Kimi Linear model at 1.4T tokens, it beat the baseline on every benchmark they ran, with GPQA-Diamond up 7.5 points and HumanEval up 3.1.The ablation is what convinced me. DenseFormer already gives every layer access to all earlier outputs, but weights them with fixed learned scalars. It scores 1.767 against a 1.766 baseline, so no gain at all. Cross-layer access on its own is worthless.The weights have to depend on the input. Swap softmax for sigmoid, and it gets worse too, because softmax makes the sources compete for a fixed budget and sigmoid doesn’t.One more result I keep thinking about. Under a fixed compute budget, the baseline’s best shape is d_model/L_b ≈ 60. With AttnRes, the best shape moves to 45, which means deeper and narrower. Same budget, and the model wants more layers. AttnRes makes depth worth more than it used to be.The MoE, and why the model is 2.8 trillionThis is the part that answers the size question, and I don’t think it’s mysterious.K3 has 896 experts and, in Moonshot’s words, “effectively activates” 16. That’s 1.8% of them. K2 had 384 and activated 8, so 2.1%. The word effectively is worth keeping, because LatentMoE routes in a projected latent space, so the literal top-k may not be exactly 16.The order of magnitude is what the argument below rests on, and that holds either way. Moonshot’s own K2 report ran this exact ablation: hold the active parameters fixed, raise the total expert count, and both training and validation loss go down, every time. K2 shipped at a sparsity ratio of 48. K3 ships at 56. They are following their own results.So the 2.8 trillion figure is mostly a statement about sparsity, not about compute. Total parameters are cheap if you barely touch them. Moonshot still hasn’t published the active count, which is the number that actually matters. Press guesses run from 40B to 100B.What makes the extra experts affordable is LatentMoE, and that one is NVIDIA’s, from Nemotron 3. The idea: before routing, project the token down from the model dimension d into a smaller latent dimension ℓ. Expert weights and all-to-all traffic both shrink byd/ℓ, usually about 4x.Then spend that saving on more experts and a bigger top-K, at roughly the same inference cost. In NVIDIA's ablation, it beat a standard MoE at matched parameters on every task. What "Stable" adds is Moonshot's, and I have no idea what it is yet.Per-Head Muon is easier to guess, and I’m fairly confident. Muon takes the update for each 2D weight matrix and orthogonalises it, meaning it evens out the update across directions.A whole matrix is its natural unit. But attention splits its projections into heads that then run independently. Apply Muon to the whole concatenated QKV, and you tie every head together through one shared factor. Split it per head, and each head gets its own update. GLM-5 already does this and calls it Muon Split. There is even a paper arguing the best answer is somewhere in between.Quantile Balancing: I can only read off the description: it sets expert allocation from router-score quantiles and removes “heuristic updates and a sensitive balancing hyperparameter”.That sounds like a shot at the DeepSeek-V3 scheme, where each expert has a bias updated by b_i = b_i + u · sign(e_i). e_i is how far off that expert's load is, and u is a rate you have to tune by hand. It's a feedback loop with a hand-set gain, so it can wobble.At 16 out of 896, a router that drifts wrecks your throughput. Reading the split straight off the score distribution avoids the loop. That's my read, not a fact.SiTU and Gated MLA I can’t tell you much about. Gated MLA is probably the sigmoid output gate from the Kimi Linear paper applied to the full-attention layers, since Kimi Linear already used one on KDA and found sigmoid beat swish. SiTU is a new activation replacing SwiGLU, and there is no paper.Does the 2.5x add up?The blog claims K3 turns compute into capability about 2.5x better than K2. Nobody showed the working, so here is mine.Both numbers I need are Moonshot’s own, which makes this the fairest check available: their claim against their own published results. Kimi Linear’s scaling law gives 1.16x over an MLA baseline. Block AttnRes, from the same team, gives 1.25x over its baseline on the Kimi Linear architecture, so the two should stack. 1.16 × 1.25 ≈ 1.45x.That leaves about 1.7x unaccounted for.Treat this as a rough check, not a proof. Those scaling runs were fitted at a few hundred million active parameters. K2 isn’t the same thing as “MLA at matched scale”. And Moonshot never defined “scaling efficiency” precisely, so I might be comparing the wrong quantities.But the gap is big enough that the direction seems safe: most of the 2.5x is not KDA and not AttnRes. It’s sparsity and data. Which is a slightly awkward finding for a launch post that led with the architecture.What it costs to serveThis is the part the coverage skipped, and it’s the reason I think the launch was competent.Prefix caching works because of an assumption KDA quietly breaks. In a normal attention layer, the KV cache is a list. Each token owns its own entry, entries never change once written, and any block boundary is a fine place to resume. Hash the block, look it up, reuse it. vLLM does this in blocks of 16 tokens.A KDA layer has no per-token entries. It has one matrix that summarises the whole prefix, overwritten every step. You can’t take “the first 900 tokens” out of it. There’s no rewind. You can only reuse a prefix if you happened to save the state at exactly that point.Source: Image by the author.A KV cache is a list of per-token entries. A KDA state is one summary of the whole prefix, and you can’t rewind a summary.So you save snapshots every so often, and the maths gets awkward fast. One KDA state is 128 × 128 = 16,384 numbers per head, about 32 KB in bf16. One token of KV is 2 × 128 = 256 numbers, or 512 bytes. Divide, and you get 64. Save a snapshot more often than every 64 tokens and your memory-saving linear attention layer is eating more cache than the KV cache it replaced.This isn’t theoretical. vLLM already deals with it for Mamba hybrids, and it’s genuinely ugly. The engine has to push the attention block size up until the attention page is at least as big as the state page.For Qwen3.5, that means 528 tokens, and since only complete blocks get cached, any prompt shorter than 528 tokens gets a 0% hit rate. The Marconi paper measured the other half of the problem: at block size 32, later requests reused 25% of KV blocks but only 0.4% of saved states. Most snapshots are never touched again.Now look at what K3 charges. Cached input is $0.30 per million tokens. Uncached input is $3.00. That’s a 10x cliff, and Moonshot says their own API runs above a 90% cache hit rate on coding work. Agentic coding is the best possible case for prefix caching, since each turn is the same transcript plus a bit more. If KDA broke prefix caching, the price sheet would fall apart.So the most important thing Moonshot shipped this week might be a pull request. They wrote KDA prefix caching and put it upstream in vLLM for day-zero support, before releasing the weights. A 2.8T model nobody can serve cheaply doesn’t help anyone.The rest fits the same pattern. Quantisation-aware training from the SFT stage, MXFP4 weights and MXFP8 activations, picked for wide hardware support rather than the best number on one vendor’s chip.Expert-parallel training with static shapes and no host sync on the critical path, because at 1.8% activation one slow expert holds up everybody. And a deployment note recommending 64 or more accelerators per node group, which tells you the all-to-all still hurts.Reading the benchmarksMoonshot’s own table has K3 behind Fable 5 on FrontierSWE (81.2 vs 86.6), HLE-Full (43.5 vs 53.3) and most vision tests, and ahead on SWE Marathon (42.0 vs 35.0), BrowseComp (91.2 vs 88.0) and OmniDocBench. Their blog says outright that K3 trails Fable 5 and GPT-5.6 Sol overall, which is more honest than these posts usually are.Some caveats before you quote any of it. Artificial Analysis found K3’s accuracy on AA-Omniscience went up from 33% to 46% over K2.6, while its hallucination rate got worse, from 39% to 51%. It knows more and makes up more at the same time. Averages hide that.ProgramBench’s author said publicly that Moonshot used a metric his team doesn’t recommend, averaging how much of each program got implemented instead of counting programs that fully work. Partial credit flatters agentic coding scores.And several Fable 5 numbers carry an asterisk for fallback, meaning refused requests were answered by Opus 4.8 instead. That isn’t a clean baseline.My read: K3 is a real frontier model, and it is not the best model you can buy. It’s also slow (Artificial Analysis clocks 62 tokens/sec against a 70 median for its price band) and wordy, though 21% less wordy than K2.6 while scoring 13 points higher.Moonshot’s own limitations section is worth your time. K3 was trained with its thinking history preserved and gets unstable if your harness doesn’t pass it back, and it will make decisions for you when a task is vague.What to watchWeights and the tech report are due 27 July. I want three things from it. The active parameter count. At 2.8T total, that single number decides whether anyone outside a well-funded lab can run this.What “Stable” means in Stable LatentMoE, and what Quantile Balancing actually computes. And the ablation that splits architecture from data. My arithmetic says the two headline changes buy about 1.45x of the claimed 2.5x. I’d like to see the rest.What K3 settles is narrower than most of the reaction suggests, and it has nothing to do with geopolitics or open weights. Hybrid linear attention works at frontier scale. That has been an open question since Katharopoulos in 2020, and every efficient-architecture paper since has ended with some version of “we expect this to hold at scale”.Moonshot spent a frontier-scale training run finding out, and then wrote the prefix caching so the answer would pay for itself.Sources: Kimi K3 blog, Kimi Linear (arXiv:2510.26692), Attention Residuals (arXiv:2603.15031), Marconi (arXiv:2411.19379), Nemotron 3 (arXiv:2512.20856), the Kimi K2 report, Artificial Analysis, and the vLLM issue tracker.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!What’s new in Kimi K3, a 2.8T parameter model. was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI