July 2026

By Davide Ciffa

Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context

poolside’s Laguna XS 2.1 ships with a DFlash speculative-decoding drafter, and lucebox runs the pair on a single RTX 3090: 296 tok/s peak at short context, and a flat 152 tok/s at 256K tokens where the full KV cache would not even fit in 24 GB. Prefill runs at ~3,500 tok/s (256K tokens in 67 s). Three optimizations got the same GPU from 22 to 152 tok/s at 256K in one pass: a drafter KV ring cache, sliding-window ring caches, and KVFlash paging. And the speculative decode is lossless: every committed token is one the model itself would produce.

TL;DR

• Decode speed stops depending on context length. Laguna XS 2.1 Q4_K_M on a 3090 holds 155 tok/s at 30K and 152 tok/s at 256K of real-code context. At the start of this work the same 256K request decoded at 22 tok/s.
• Prefill: 256K tokens in 67 seconds (~3,500 tok/s), down from 411 s. Using batched pooled prefill + ring attention; the remaining cost is the MoE expert GEMMs.
• Short context: 242 tok/s mean, 296 peak, HumanEval-10 at 230 tok/s, GSM8K answers 10/10. Speculative decoding is verify-exact, so quality is the model’s own at any speed.
• Two new engine optimizations, both model-agnostic. The DFlash drafter now keeps a context-KV ring cache (draft step 9.9 → 2.4 ms, flat at any context), and sliding-window layers attend a 2K position ring instead of the whole KVFlash pool (verify 19.4 → 13.7 ms, ~1 GB VRAM freed).
• Built with poolside’s drafter. Acceptance stays ~56% on long real-code prompts;

The model, fast MoE with a drafter in the box

Laguna XS 2.1 is poolside’s 33B MoE: 40 layers, fine-grained experts, 2048-token hidden size, and a 3-in-4 pattern of 512-token sliding-window attention layers. It ships with an official DFlash drafter: a 5-layer block-diffusion head that reads the target’s hidden states and proposes 16 tokens per step. lucebox verifies those proposals in one batched target forward and commits only tokens the target itself would emit: speculative decoding with exact greedy equivalence, which is why every number below comes with unchanged output quality (GSM8K 10/10).

Short context was already fast: 242 tok/s mean, 296 peak on a 3090 after the earlier rounds of work (quantized q4 drafter on HuggingFace, fused decode loop, CUDA-graph replay everywhere). Long context was a bigger problem: at 256K tokens the same server decoded at 22 tok/s and took 20+ minutes to prefill. This post is about closing these gaps:

Three optimizations

The 22 tok/s at 256K was not one bottleneck but three, and each fix is now a default in the lucebox engine.

1. KVFlash paging for the full-attention layers (background). The pool holds 8K of the 256K context on the GPU; cold 64-token chunks page to host RAM bit-exact, and a drafter scores which chunks deserve residency. Prefill runs through the pool in 1024-token batches with CUDA-graph replay. This is what makes 256K fit at all: the full KV cache would need ~20 GB next to 19 GB of weights.

2. A context-KV ring cache for the drafter. DFlash drafters re-encode their feature window (up to 4K tokens of target hidden states) every step. That cost ~10 ms per step once the window filled, on every DFlash model family, hidden until now behind slower targets. The drafter’s K/V now live in a position-indexed ring that only encodes newly committed tokens: draft step 9.9 → 2.4 ms, constant at any context.

  per-step time at 256K context, RTX 3090

  before      draft 9.9 │ verify 19.4 │ heads 0.8   ≈ 22 tok/s*
  + draft-KV  draft 2.4 │ verify 19.5 │ heads 0.8   → 109 tok/s
  + SWA rings draft 2.4 │ verify 13.3 │ heads 0.8   → 152 tok/s

  * session start also paid thermal taxes

3. Ring caches for the sliding-window layers. 30 of Laguna’s 40 layers attend a 512-token window, but under paging they flash-attended the whole 8–16K pool span with 97% of it masked out, so most of the attention FLOPs in both prefill and decode were masked-out work. Window layers now keep a small position ring (slot = pos mod 2048) and skip the pager entirely: verify 19.4 → 13.7 ms, prefill −27%, ~1 GB of VRAM freed, acceptance unchanged.

Speed and quality

All runs: Laguna XS 2.1 Q4_K_M with the Q4_K_M DFlash drafter, single RTX 3090, real-code prompts, 256-token timed generation. “Start of this work” is the first working KVFlash port of laguna (32K pool, no drafter cache, no rings); “after” is the shipping default.

Decode is flat from 30K to 256K because every step touches a pool-sized cache plus 2K-token rings, never the full context. Acceptance drops from 81% (short prompts) to ~55% (long real code). That is the drafter’s shorter training window showing, not the paging: pooled attention actually scores slightly better acceptance than the full cache at equal length. Since verification is exact, acceptance only costs speed, never correctness: answers stay 10/10 and 256K deep-context answers are correct.

Prefill deserves its own note. At 256K it now runs at ~3,500 tok/s; the profiler says the remaining cost is two-thirds MoE expert GEMMs and one-third pool attention. The next prefill gains come from feeding the experts bigger batches (1024-token batches alone were −19%).

Usage

The full stack below (speculative decoding, KVFlash paging, the drafter ring cache and the SWA rings) is the default path. One command:

# build the server (CUDA 12+, sm_86+)
git clone https://github.com/Luce-Org/lucebox-hub && cd lucebox-hub/server
cmake -B build -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j

# target (poolside official), our q4 drafter, and the PFlash residency scorer
huggingface-cli download poolside/Laguna-XS-2.1-GGUF Laguna-XS-2.1-Q4_K_M.gguf --local-dir models
huggingface-cli download Lucebox/Laguna-XS-2.1-DFlash-GGUF laguna-xs21-dflash-q4.gguf --local-dir models
huggingface-cli download Qwen/Qwen3-0.6B-GGUF Qwen3-0.6B-Q8_0.gguf --local-dir models

# the measured configuration: 296 tok/s short context, 152 tok/s at 256K
./build/dflash_server models/Laguna-XS-2.1-Q4_K_M.gguf \
    --draft models/laguna-xs21-dflash-q4.gguf \
    --prefill-drafter models/Qwen3-0.6B-Q8_0.gguf \
    --max-ctx 262144 --kvflash 8192 --chunk 1024

# pool sized from free VRAM instead of fixed:
./build/dflash_server models/Laguna-XS-2.1-Q4_K_M.gguf \
    --draft models/laguna-xs21-dflash-q4.gguf \
    --prefill-drafter models/Qwen3-0.6B-Q8_0.gguf \
    --max-ctx 262144 --kvflash auto

--prefill-drafter is the same Qwen3-0.6B PFlash drafter every lucebox family uses: it compresses prompts and scores which KV chunks stay resident under paging (without it the pool falls back to recency-only LRU; see the KVFlash post). It is auto-probed from the model directory or /opt/lucebox/models/drafter/ if the flag is omitted. Both model downloads are official: the target from poolside/Laguna-XS-2.1-GGUF, the quantized drafter from Lucebox. An 8K pool is the measured sweet spot on 24 GB (152 vs 146 tok/s against a 16K pool, with slightly tighter recall); --chunk 1024 buys the last 19% of prefill. Startup logs confirm the path: [draft-kv] ctx-KV ring active and [laguna][swa-ring] SWA layers on 2048-row position rings. Envs for A/B: DFLASH_DRAFT_KV=0, DFLASH_LAGUNA_SWA_RING=0.

Bottom line

A $1000 GPU now runs poolside’s Laguna XS 2.1 at 296 tok/s peak, and holds 152 tok/s at 256K tokens of context, a context that does not physically fit on the card. Two new optimizations are model-agnostic and are available in every DFlash family lucebox serves.

Source: github.com/Luce-Org/lucebox-hub (drafter ring cache in server/src/common/dflash_draft_kv.h, SWA rings in the laguna backend, pager in server/src/common/kvflash_pager.h). Quantized drafters: Lucebox/Laguna-XS-2.1-DFlash-GGUF. Numbers measured on an RTX 3090 24 GB, Laguna XS 2.1 Q4_K_M, Q8_0 KV, July 2026, temperature/clock-certified.

Related

• Luce KVFlash: 256K context with 72 MiB of KV on the GPU
• Laguna XS.2: the first MoE PFlash target
• Luce Spark: a 35B MoE on a 16 GB GPU, without the offload tax
• DFlash on ggml: up to 207 tok/s Qwen3.5-27B on a RTX 3090