June 2026

By Davide Ciffa

Luce KVFlash: 256K context with 72 MiB of KV on the GPU

Long context has a second memory bill nobody budgets for: the KV cache. On Qwen3.6-27B at 256K tokens it costs 4.6 GiB of VRAM and drags decode down to 13 tok/s, because every new token reads the whole thing. Luce KVFlash keeps a small pool of KV on the GPU, auto-sized to your VRAM, and pages cold 64-token chunks to host RAM, bit-exact and recallable. Decode holds a flat 38.6 tok/s from 64K to the native 256K on an RTX 3090, 2.9x the full cache at 256K, with 72 MiB resident and benchmark accuracy unchanged. One flag, every model family lucebox serves.

TL;DR

• Decode speed stops depending on context length. Qwen3.6-27B on a 3090: the full cache decays from 27.8 tok/s at 64K to 13.1 at 256K. KVFlash holds 38.6 tok/s at all three, because each token reads a pool-sized cache instead of a context-sized one. Prefill gets faster too: 2.8x at 256K (355 s vs 999 s).
• 99.2% of attention KV leaves the GPU. At 128K the attention KV drops from 2304 MiB to 18 MiB with a 1K pool. At 256K you hold 72 MiB resident plus 4.2 GiB of ordinary host RAM. Evicted chunks come back bit-exact; this is paging, not compression.
• The model still finds what it needs. With the PFlash drafter scoring which chunks matter, needle recall holds at 14-16/16 from 8K to 256K at 6-9% residency. On HumanEval, GSM, MATH, and agent tasks: 36/36 with KVFlash vs 36/36 with the full cache.
• Every model family, one flag. --kvflash auto sizes the pool from your VRAM and works on qwen35, qwen35moe (Spark hybrid included), laguna, and gemma4. Smoked through live eviction at 37 to 137 tok/s depending on the model.
• The idea comes from DeepSeek. KVFlash is a FlashMemory-style lookahead sparse attention (arXiv 2606.09079), with two twists: the PFlash drafter replaces their trained indexer, so there is nothing to train, and a fixed slot pool replaces their threshold, so VRAM is capped by construction.

The problem: the cache that grows whether you like it or not

Weights are a one-time bill. The KV cache is a tax on every token of context: on Qwen3.6-27B with Q8_0 KV it is about 18 KiB per token, so a 256K context costs 4.6 GiB of VRAM before the model generates anything. On a 24 GB card that already holds 15 GiB of weights, the cache is what decides how much context you can actually afford.

The memory is half the problem. Decode reads the entire cache once per generated token, so attention time grows linearly with context. You can watch it happen:

The standard fixes all trade something you want. Quantizing the KV (lucebox already runs Q8_0 by default) shrinks the per-token cost but keeps the linear growth. Sliding windows cap the cache but forget everything outside the window. Compression schemes drop tokens for good. What you actually want is the full context kept somewhere cheap, with the GPU only holding the part attention is about to use.

How it works: page the cache like an OS pages memory

KVFlash allocates the attention KV tensors at a fixed pool size, say 4K slots, instead of the 256K maximum. A pager maps logical token positions to physical pool slots at 64-token chunk granularity. When the pool fills, the coldest chunk's quantized rows are copied to a host backing store (0.6 ms per chunk) and its slots are reused. Attention runs over the pool with a slot-validity mask, so evicted positions are excluded exactly, not approximately.

  logical context  [0 ......................... 256K)
                       │ 64-token chunks
                       ▼
  GPU pool  [4K slots]   hot chunks, sinks, recent tail
  host RAM  [the rest]   cold chunks, bit-exact, recallable

Two properties make the relocation legal. RoPE is baked into the K rows at write time, so a row means the same thing in any physical slot. And paging moves raw quantized bytes, so a chunk that leaves and comes back is bit-identical to one that never left. The first chunk (attention sinks) and the trailing window are never evicted.

The interesting question is which chunks deserve the slots. Two policies ship:

• LRU (no dependencies): recency decides. Good for chat and agent loops where attention is local; useless for "the answer was 200K tokens ago".
• Drafter-scored (the default, with PFlash): the same Qwen3-0.6B drafter that compresses prompts for prefill re-scores every chunk against the generation tail every 64 decoded tokens, and the pager repages the pool to the top-scoring set. The drafter acts as the paper's "memory indexer", except ours was already shipping and needs no training. A recall of an evicted chunk is a 0.6 ms copy.

Memory: the part of the cache you hold goes flat

Attention KV in VRAM by context length, Qwen3.6-27B, Q8_0 KV, pool of 4K tokens:

At 256K the ledger reads: 72 MiB of KV on the GPU, 4.2 GiB in host RAM, and the roundtrip is bit-exact. Paging costs are small enough to ignore in practice: 0.6-1.3 ms per 64-token chunk, hidden behind decode steps that take 26 ms each, and a full repage during reselect costs about 1% of decode time.

Speed and quality, measured together

Here is what the 128K row feels like. Same model, same card, same 128K prompt, same 240-token answer; the only difference is the flag. Prefill is shown as a labeled timelapse (335.9 s vs 177.8 s); the decode race runs at the measured rates in real time:

Numbers without a quality column are how KV tricks usually hide their bodies. Full A/B on Qwen3.6-27B Q4_K_M, RTX 3090, needle-in-haystack prompt at depth 25%, 240-token timed generation, KVFlash pool of 4K with the drafter policy:

Read it honestly: at 32K KVFlash is slightly slower (29.0 vs 32.8) because a 32K cache is still fast to read and the pool adds a rescore. The crossover is around 64K, and from there the gap only widens: 1.4x at 64K, 2.0x at 128K, 2.9x at 256K. Recall gives up 1-2 needles out of 16 at 6% residency; the policy decides what those misses cost, which is why the policy is pluggable.

The ablation that justifies the drafter: recency-only LRU scores 0/16 the moment the needle falls outside its tail window. Relevance scoring is what makes 6% residency usable for retrieval, not the paging itself:

On benchmarks where the answer depends on the whole prompt rather than a needle, nothing moves: HumanEval 10/10, GSM 10/10, MATH 10/10, agent tasks 6/6, identical pass rates to the full cache, with a base-vs-base control confirming the harness itself is deterministic. Speculative decoding also runs directly on the pool, at acceptance parity (15.4-15.6% vs 15.3%).

Every model family lucebox serves

The pager core is architecture-blind; each backend routes its KV writes and masks through it. Same flag, all four families, all smoked through live eviction with a deliberately small 1K pool:

One caveat worth stating plainly: the drafter scores Qwen tokens, so on laguna and gemma4 it runs through a cross-tokenizer bridge (detokenize the target's history, re-tokenize for the drafter, score, map back by character spans). It is the default there too and strictly better than LRU, but not yet tuned to the Qwen-native 14-16/16, so treat long-range recall on those two as functional but not yet dialed in. They get the memory cap and the flat decode either way.

Usage

# recommended: drafter-scored residency, pool auto-sized from your VRAM.
# pass --prefill-drafter so the drafter is guaranteed (no silent LRU fallback).
dflash_server model.gguf --max-ctx 262144 --kvflash auto \
    --prefill-drafter qwen3-0.6b.gguf

# drop the path to auto-probe (model dir, drafter/, draft/, /opt/lucebox/models/drafter/);
# falls back to LRU if none is found, so check the banner reads policy=drafter
dflash_server model.gguf --max-ctx 262144 --kvflash auto

# explicit pool size, recency-only LRU
dflash_server model.gguf --max-ctx 262144 --kvflash 8192 --kvflash-policy lru

With --kvflash auto the pool sizes itself from free VRAM and the drafter loads on demand; pass --prefill-drafter to be sure it is used, since auto-probe falls back to LRU if no drafter is found. If you'd rather size by hand: without a drafter, LRU is recency-only memory, so size the pool to prompt plus generation headroom or it will evict the question itself; with the drafter, 25% of the expected context is a conservative default and 6-9% is measured safe for retrieval work. Memory-dense tasks that genuinely need every token at once (multi-round co-reference over the whole context) are the paradigm's honest limit, shared with the paper: size the pool up for those.

Bottom line

The KV cache was the last part of long context that scaled linearly with VRAM and decode time. Paged through a fixed pool with relevance deciding residency, it stops doing either: 256K context on a 24 GB card with 72 MiB of KV resident, decode flat at 38.6 tok/s where the full cache sinks to 13, prefill 2.8x faster, and the benchmark column unchanged. It composes with PFlash (which keeps the prompt small) and Spark (which keeps the experts small), because they manage different memory. On a local-inference PC the practical effect is simple: context length stops being the thing you ration.

Source: Luce KVFlash on github.com/Luce-Org/lucebox-hub (docs and measured results in optimizations/kvflash/, core in server/src/common/kvflash_pager.h). Numbers measured on an RTX 3090 24 GB, Qwen3.6-27B Q4_K_M, Q8_0 KV, June 2026. Design follows FlashMemory lookahead sparse attention, arXiv 2606.09079.

Related

• PFlash: 10x prefill speedup over llama.cpp at 128K on a RTX 3090
• Luce Spark: a 35B MoE on a 16 GB GPU, without the offload tax
• DFlash on ggml: up to 207 tok/s Qwen3.5-27B on a RTX 3090