Inference Systems Infrastructure Notes (6): HiSparse — Treating HBM as a Cache to Break the Long-Context Capacity Wall

0. The One-Line Thread

The earlier posts (sys3 / sys5) asked “how do we make KV smaller and read less of it”; this one tackles the half that is still too big even after shrinking — demoting GPU HBM from “where KV lives” to “a cache over KV.”

HiSparse (Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management, Stanford MAST Lab, arXiv:2608.07009, Aug 2026) makes a single argument: if top-k sparse attention reads only k KV entries per step, why should HBM hold L_ctx of them? Move the full KV history to host DRAM, keep a small fixed-size hot cache on the GPU, and add one fused CUDA kernel that does hit detection, LRU replacement and host-to-device fetch in a single launch — and you raise long-context concurrency while keeping the output bit-for-bit identical, token by token.

It is merged into upstream SGLang and evaluated across three sparse-attention families (DSA, NSA, Quest) on H200, B200 and GH200, with up to 4.7x higher peak generation throughput on long-context workloads.

1. Capacity Wall: Sparse Attention Saved FLOPs, Not Memory

Start with the problem. Top-k sparse attention (NSA, DSA, Quest, and friends) sells cheap compute: at layer ℓ and step t it selects only k historical positions S_t^(ℓ) — typically a few thousand, not the full context length L_ctx.

But serving stacks carry a hidden assumption that drags everything back:

Any past token may be selected by the indexer at some future step.

To avoid missing one, systems simply keep the entire KV cache resident in GPU HBM. The memory bill still grows linearly with L_ctx — so memory runs out long before compute does. That is the capacity wall the paper names.

The numbers are blunt:

Note the mismatch: the compute side already reads only k entries, while the memory side still pays for all of L_ctx. That is exactly what HiSparse removes — HBM footprint should scale with k, not with L_ctx.

The hard part is that sparse selection is dynamic: S_t^(ℓ) changes every step and every layer, so you cannot statically “compress it once and freeze it” the way MLA does.

2. Two-Level Hierarchy: Authoritative Copy on Host, Hot Cache on GPU

HiSparse never touches model logic — it only changes where KV records live:

Three components, three jobs:

1. Host KV pool — KV records produced during prefill go straight into pinned host memory. This is the only authoritative copy.
2. GPU hot cache — B slots per request-layer pair holding recently selected KV records. To guarantee the current attention can always proceed, B ≥ k.
3. Metadata — compact page tables on the GPU map a logical token position to a physical slot or mark it host-only, plus recency bits for LRU.

Per-request HBM consumption becomes:

HBM usage = N_l x B x W_KV x s

where N_l is layer count, W_KV is KV elements per token, and s is bytes per element. L_ctx is gone — that is what “decoupling decoding throughput from GPU memory capacity” actually means.

2.1 How Slots Are Actually Allocated (DRAM to HBM Mapping)

Three points worth memorizing on their own:

1. Allocation granularity is request x layer, not request. Different layers select different entries, so each layer needs its own set of slots — which is why N_l appears in the formula.
2. DRAM and HBM are decoupled by the page table. The DRAM side is addressed contiguously by logical position; HBM slots have no fixed relation to logical position and simply hold “the recently selected ones,” with all mapping in the page table.
3. B is a constant, independent of context length. This is exactly where the capacity wall comes down: as context grows from 128K to 1M, the DRAM side grows linearly (fine, host memory is large) while the HBM side does not move at all.

3. RESOLVE: Five Stages in One Launch

Hierarchical caching is an old idea. The hard part is making a miss cheap enough to tolerate, because sparse selection access patterns are scattered, and a conventional CPU-side paging path exposes the full latency.

HiSparse writes a single fused CUDA kernel named RESOLVE, launched once per sparse layer, doing five things in parallel across GPU threads:

GPU-assisted IO is the real technical contribution here. Letting GPU threads read host memory directly saturates PCIe or NVLink bandwidth even for scattered access, bypassing the long “CPU notices a fault, notifies, then copies” path. And the whole flow runs inside the decode CUDA Graph, so dynamic control flow never breaks graph capture.

4. Locality: Why a Small B Is Enough

Every cache lives on locality, and HiSparse shows sparse selection has two exploitable properties:

• Temporal locality — consecutive decoding steps re-select many of the same tokens;
• Cross-layer correlation — different layers tend to attend to nearby regions of the context.

Measured miss rates on GLM-5.1 (k = 2048) make the point:

Exact Layer-Wise Prefetching

There is a sharper trick. Some models (for example GLM-5.2) share indexer output across groups of layers — once an anchor layer fixes its selection, the system immediately knows the selections of the layers that share it.

HiSparse turns this into exact layer-wise prefetching:

1. The anchor layer computes its selection, which yields the shared layers’ miss plans deterministically;
2. A background copy-only kernel replays those future layers’ miss plans;
3. Host-to-device transfer then overlaps with the current layer’s compute.

Result: roughly half of the remaining IO latency is hidden. Note the word exact — this is not a heuristic guess but deterministic knowledge, so it introduces no correctness risk.

5. Why It Is Exact and Indexer-Agnostic

These two properties are why it can go straight into production:

• Exact — it changes only the physical placement of KV records: no model surgery, no change to attention math, no approximate recall. Model output is unchanged token by token, which separates it fundamentally from “trade memory for approximate retrieval” schemes.
• Indexer-agnostic — it does not care how you picked those k positions. The paper evaluates DSA, NSA and Quest and all three benefit.

In short, HiSparse is a general-purpose memory backend for sparse attention: if your model already has a sparse indexer, it just plugs in.

6. Measured Results

That last row matters: host-device IO is the only, and the entire, price of this design. Which means link bandwidth directly sets the payoff:

Link sensitivity: KV fetch time on GH200 (NVLink-C2C) is nearly 4x lower than on H200 (PCIe Gen5), allowing a smaller GPU cache and higher concurrency.

7. When Not to Use It (Limits and Preconditions)

This section matters more than the speedups:

1. It assumes host DRAM is much larger than GPU HBM. On Grace-based GB200 / GB300, CPU and GPU share unified memory and the host side has no capacity advantage — the whole offload-for-capacity argument does not hold. The paper lists this as a fundamental limitation.
2. It relies on GPU-assisted IO reaching near-link bandwidth for scattered fetches. That is borrowed from the authors’ Strata work and is not independently benchmarked in this paper. Misses cost noticeably more on PCIe Gen5 than on NVLink-C2C.
3. It buys concurrency and capacity, not single-request speed. Per-token latency is merely comparable; it will not make one request finish faster.
4. It does nothing for dense-attention models. You need a sparse indexer first — without top-k selection there is no footing for “only k entries must be resident.”
5. Payoff varies with load. At low concurrency or short context the baseline never hits the capacity wall, and the hierarchy only adds an IO hop.

8. The One-Line Thread, Completed

Stack sys3, sys5 and this post together and the KV story is whole:

MLA makes each token store smaller; NSA / DSA / Quest read only the important k entries; CSA compresses tokens first and then does sparse reads; HCA compresses very long history into a short summary and reads all of it — while HiCache and HiSparse answer a different question: once KV exists, which tier of the memory hierarchy should hold it?

One more distinction:

• HiCache asks “should KV sit on GPU, CPU, or even lower tiers” — a storage-tier question;
• HiSparse asks “under sparse decoding, how much must actually be resident on the GPU” — making the capacity bill scale with k rather than L_ctx.

Cutting compute (sparse attention) and cutting capacity (compression plus hierarchy) are orthogonal axes. HiSparse sits at the far end of the second one, and is currently the closest to production (it is in upstream SGLang).

9. Links to Earlier Posts

• Full KV compression landscape (MHA to HiCache) — see sys5-attention-evolution-kvcache
• MSA / CSA / HCA: three attention redesign routes — see sys3-attention-evolution
• HiCache and the NUMA x PCIe x NIC data path — see sys1-pcie-numa-nic-hicache
• SGLang internals (prefix reuse and throughput optimization) — see fw3-sglang-internals
• HiCache / Mooncake KV reuse and compositional correctness — see tr2-vllm-sglang-20260807

Appendix: Paper

HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Ziyi Xu, Ruiyang Ma, Christos Kozyrakis
Stanford MAST Lab - 2026-08-07 - arXiv:2608.07009
Code: merged into upstream SGLang

This post is a close reading and engineering interpretation of the paper. It is not technical or investment advice. Figures are taken from the paper and its alphaxiv reading page (retrieved 2026-09-07).