SGLang fits FULL, SWA, and Mamba reuse rules into a single radix tree — hits are decided by component voting, migration handled by HiCache's same-name transfer, and eviction guided by session awareness. One tree, rules delegated to components.
End of analogy. The real difference: the tree only assigns coordinates to each prefix segment, while FULL, SWA, and MAMBA each act as components defining their own reuse boundaries, with voting determining the deepest node that passes all checks.
Three reuse rules — how does one tree hold them all?
The three reuse boundaries are different, and forcing them under the same prefix is wasteful or even overreaching. The real difference: SGLang keeps just one tree, and extracts the rules out of it.
Hybrid models stuff full-attention KV, sliding-window KV, and recurrent state into the same request, but shared token prefixes (the smallest unit sequences of text the model reads in) can't share reuse boundaries. Full-attention KV is valid across the entire prefix; sliding-window (Sliding Window Attention, SWA) KV is only valid within the tail window; Mamba (a recurrent layer architecture using fixed-size state instead of full KV, MAMBA) recurrent state is only usable at exact checkpoint points. Earlier implementations wrote a separate cache class for each combination and stacked orthogonal capabilities like HiCache on top — cache classes exploded in permutations, and matching, insertion, locking, and eviction logic all got duplicated.
Unified Radix Cache separates these two concerns. A radix tree (a prefix tree indexed by token sequence, radix tree) organized by token prefix assigns a unique coordinate to each prefix segment, and FULL, SWA, and MAMBA each attach as a TreeComponent.
How is the reuse boundary determined? By component voting, not a single rule. FULL traverses the full path, SWA traverses the tail window, MAMBA only picks up exact checkpoints. During matching, UnifiedTreeCore walks the FULL path and every visited node is a candidate boundary; each active component casts a validator vote, and the deepest node that all votes pass is considered safe. DeepSeek-V4 uses FULL+SWA, Kimi-K3's KDA (a linear attention variant using fixed-size state for recurrence) runs FULL+MAMBA, and Inkling packs all three components on the same tree.
New model families can directly compose existing component combinations; to add a new reuse rule, attach a new TreeComponent — no need to build another tree.
The tree only handles traversal, rules go to component voting
Who decides the boundary? Not the tree — it's the three components attached to the nodes. FULL, SWA, and MAMBA each define their own "reusable boundary," and when opinions clash, a vote decides. A new model just picks a component combination — no need to build another tree.
SGLang's Unified Radix Cache lets the radix tree only handle prefix coordinates, while FULL, SWA, and MAMBA each decide their own boundaries — the tree stays as one.
So how is the boundary found? By voting. UnifiedTreeCore walks the FULL path, and every visited node is a candidate boundary. FULL passing isn't enough — each node also gets handed to every active component's validator: sliding window checks tail-window continuity, recurrent state checks for an exact checkpoint.
If any component vetoes, swap and try again. The deepest node that all components agree on is the reusable boundary. After the walk, the MatchResult goes to each component's finalizer for cleanup — for example, MAMBA checkpoints must be copied from the shared copy into a private slot before writing.
The cost of voting is that nodes may end up "half-empty." When old slots get evicted, the SWA component leaves behind tombstones (grave markers indicating the position is now empty), while the radix node itself stays put; after MAMBA's private copy is consumed, the node may similarly have empty slots. These empty slots only get cleared when a component reactivates or the node is invalidated.
Data from different components under the same prefix may land in mismatched positions, but the tree's topology stays stable — components pick up their own portion and keep working.
The rest of the lifecycle is wired up through hooks. During matching, create_match_validator decides whether a node is usable; during splitting, redistribute_on_node_split repositions KV; during insertion, handle_node_insertion handles overlap; during eviction, dispose_node determines cross-device release order. The tree core only handles passing and positioning; rule details go back to the components.
Onboarding a new model is compressed into two steps: pick a component combination, or add a new TreeComponent. Start by choosing from existing FULL / SWA / MAMBA combinations and run the same prefix coordinates; only when a new rule appears (e.g., another variant of recurrent state（recurrent state (a small memory unit the model uses to compress long context)）) does the TreeComponent interface get touched — no new trees will grow. Subsequent HiCache tiered migration and eviction policies hook into the same tree's same set of component hooks, with no need to reassemble an entire caching stack. New model families compose these capabilities without introducing another cache tree.
Three attention types sharing one tree — who decides?
SGLang attaches three reuse rules to the same radix tree（radix tree (a prefix tree indexing prefixes by token sequence)）: FULL（FULL (valid across the entire prefix)）, SWA（SWA (sliding window attention, covering only the tail segment of tokens)）, and MAMBA（MAMBA (recurrent state layer, only valid at exact checkpoint positions)）. Candidate reuse points are validated component by component; any veto triggers a fallback, and the nearest node all components agree on is the final answer.
Figure 2 shows this in action. UnifiedTreeCore walks the FULL path to n4, but n3 and n4 are rejected by at least one component, so n2 becomes the reuse boundary all three accept. Reuse depth is the voting result, not the traversal depth.
Fully compound models like Inkling, which mix all three attention mechanisms simultaneously, don't need an extra tree implementation.
After matching, finalizers prepare the result — state may be copied once a shared MAMBA checkpoint is claimed, keeping concurrent writes from corrupting shared state.
Eviction starts recognizing people: active sessions stay first
Fitting different reuse rules into the same tree is just the first step. The tree also has to tell who's still online and who's already gone. The next challenge is eviction.
A session still actively chatting with the model and one that's been silent for ten minutes have vastly different cache value — the former's next request is very likely to continue using this prefix, while the latter may be discarded entirely at any moment.
SGLang's answer is session-aware eviction: bind cache entries to the session that produced them, and during eviction prefer to keep entries from active sessions — without hard-pinning. The original text puts it plainly:
On the SWE-bench workload, the session-aware version achieves 2.9% to 16.6% lower TTFT (time to first token) compared to plain HiRadixCache + LRU. What do those two numbers represent?
TTFT is time to first token — the lower it is, the faster the user sees the first character after sending a request; LRU is the classic "least recently used" eviction algorithm, kicking out entries that arrived earliest. The session-aware version delivers lower latency under both extreme workload conditions, showing its advantage isn't limited to a single request pattern.
The gains don't come from eviction alone. SGLang is also migrating the tree core to Rust, and the prototype further reduced TTFT by up to 42% in rounds 176 to 200 of the sliding window benchmark. This test range corresponds to the later portion of extremely long conversations, where the tree structure's own overhead starts becoming the bottleneck.
Each radix node (a branch point in the tree) access in the Python implementation incurs interpreter cost — the longer the prefix and the deeper the node, the higher the overhead. The Rust version compresses this down.
Putting these three things together: components decide whether reuse is possible, HiCache decides which layer the cache lives on, session-aware eviction decides who gets kicked first, and the Rust core decides the traversal cost of large trees. None of them alone can claim credit for the TTFT improvement.
Signals worth watching in the next phase:
- Whether session-aware eviction can hold onto that 16.6% lead under production long-context workloads.
- Whether the Rust tree core stays stable at that 42% figure when all three components — FULL+SWA+MAMBA — are fully active.
- If either side's numbers drop, it means the current "soft eviction + cross-language migration" combo isn't yet ready for full production rollout.
Watch one tree vote, then decide whether to use it
Before getting hands-on, you need to know what FULL, SWA, and MAMBA each handle, and how session-aware eviction and HiCache's three-tier migration hook into the same tree.
-
Check the component combination against your model.
Unified Radix Cache is a single token-keyed tree — FULL is always present; if the model has sliding window attention (an attention mechanism that only looks at the most recent N tokens), there's an additional SWA component; if it has Mamba-type recurrent layers (a layer that compresses history into fixed-size state), there's an additional MAMBA component.
What to check: Does your model have SWA or Mamba layers? This determines how many components hang on the tree. If the combination doesn't match, session-aware eviction and the L3 external layer won't be usable.
-
Distinguish KV cache hits from component voting.
SGLang's end-to-end reuse is on the KV cache (storing intermediate results computed during inference for reuse, avoiding redundant computation), not the tree itself. Two requests sharing the same system prompt will show a noticeably lower TTFT (time to first token) on the second one — but this only proves KV hits, not that all three components voted. The former means the cache works; the latter means component validation passed. They're different things.
What to check: Run two identical requests with a shared system prompt and measure TTFT. A drop confirms KV reuse, but you haven't verified component voting yet.
-
Understand the voting flow in Figure 2.
The FULL path reaches n4, but n3 and n4 fail at least one component's validation, so the reuse boundary stops at n2. The tree trunk walks to the bottom, but the reuse boundary may not.
What to check: Trace the figure's paths — which nodes pass all three components' checks, and where does the boundary actually stop? Understand that first, then talk about whether to use it.
-
Verify HiCache's cross-tier migration.
The L3 external layer keeps DeepSeek-V4-Flash's late-stage hit rate at around 98% and Inkling-Small's at around 96.8% across multi-round benchmarks. HiCache controls whether a component's carrier lives on GPU L1, Host L2, or external L3 — unchanged component names mean identity is migratable; after crossing tiers, the same prefix is still recognized.
What to check: The original text doesn't provide a reader-reproducible multi-round benchmark script, so this item can only watch for whether the team later releases scripts and weights — there's currently no hands-on re-testing path.
-
Observe session-aware eviction.
Under the SWE-bench (a benchmark measuring a model's ability to solve real software engineering tasks) workload, the session-aware configuration achieves 2.9% to 16.6% lower TTFT than plain HiRadixCache + LRU (least recently used eviction strategy).
What to check: Get the same SWE-bench task flow and log TTFT distributions with session-aware on and off — see whether active sessions' prefixes are preferentially kept. The original text doesn't provide a ready-made re-test script here; once the team releases one, run the method above and compare.
Two things haven't happened yet: the Rust tree core prototype achieves up to 42% TTFT reduction in rounds 176 to 200 on the sliding window benchmark — that's engineering progress, not something you can run now.
This article is based on the original LMSYS Org Blog post (2026-08-11). Numbers published by the vendor (benchmarks, reduction percentages, etc.) are official figures and, unless otherwise noted, have not yet been independently verified by third parties.