September 2026

By Umberto Breglia

Continuous batching: Qwen3.8-27B at 301 tok/s across five clients, up to 1.97× llama.cpp

Lucebox now serves several requests at once and refills each slot the moment it frees up. On one R9700, Qwen3.8-27B with DFlash2 goes from 106.6 tok/s at one client to 300.9 at five, up to 1.97× llama.cpp at equal concurrency. Here is how the scheduler, stable slots, and paged KV make that work, for Qwen3.8 DFlash2 and DeepSeek V4 Flash.

A local inference server can receive concurrent requests from several people or from a single user running multiple subagents. Those requests rarely line up: one is still processing a long prompt, another is generating a response, and a third has just finished. Continuous batching lets the server reconsider that group after each model step, instead of waiting for the entire batch to finish.

Lucebox uses a shared request scheduler for any model backend that implements SeqEngine. New backends can use the same admission, scheduling, and request lifecycle by implementing that interface. Each backend chooses how to batch its model operations, so its prefill behavior and performance curve can differ.

Why static batching is not enough

Static batching works well when requests arrive together and have similar prompt and response lengths. In practice, requests arrive at different times and take different amounts of work.

One request may be decoding token 200. Another may arrive with a 3,000-token prompt. A third may finish while both are running. If the server builds a batch once and waits for every member to complete, finished requests leave idle slots until the longest request completes. If it serves every request independently, the GPU has to run through the model separately for each request.

Here is an illustrative schedule for a backend that can mix prompt chunks with decode rows. Chunk sizes are examples, not fixed defaults:

t0  prefill(A, B)
t1  decode(A, B) + prefill(C[0:512])
t2  decode(A, B) + prefill(C[512:1024])
... finish C's remaining prompt
t3  decode(A, B, C)
tN  B finishes -> reuse its slot and KV blocks for another sequence

The server updates the batch after each step to include requests that are ready to run.

More throughput does not mean less waiting

Generating a token requires another pass through the model's layers. With several requests in a batch, a projection can use the same weights for several rows of input. That gives the GPU more work per weight read and can improve aggregate throughput.

The larger step can still take longer than a one-request step. Five users sharing a server that produces 250 tok/s do not each receive 250 tok/s. Prompt length, scheduling, and speculative acceptance also affect their experience. Mixture-of-experts models add another dependency: which experts the requests select and how that work is distributed.

One scheduler, model-specific execution

The scheduler selects work without knowing the model's tensor layout. The HTTP server owns requests and streaming connections. The scheduler owns the FIFO queue, admission, fairness, cancellation, and the next StepPlan. The plan names the slots that decode, the prompts that advance, and the amount of prefill that fits in this iteration.

SeqEngine owns the model-specific half. An admitted request receives a stable slot containing its block table and persistent sequence state. The engine lowers the selected prompt chunks and decode rows into the graph layout its backend understands. It executes that plan and returns a StepResult keyed by slot ID. The scheduler maps those slots back to requests and delivers each result to the correct client.

Each backend stores the state its model needs in the sequence slot. This can include paged attention KV, recurrent state, convolution history, or other model data. The HTTP scheduler does not need to understand any of those layouts.

When a client reads slowly, its generated bytes are buffered per request while the scheduler continues processing other sequences. Finished and cancelled requests retire their slots and return their KV blocks to the shared pool.

Keeping track of each request

As requests join and leave, their positions in the active batch can change. Each request therefore needs a stable identity that keeps its cached history and sampling state attached to it across model steps.

Each admitted request gets a physical sequence slot:

request        -> scheduler-owned lifetime
sequence slot  -> persistent model and sampling state
block table    -> logical token positions -> physical KV blocks
step plan      -> work selected for the next model step

The compact batch might contain slots [0, 3, 7] on one step and [0, 2, 7] on the next. Explicit row-to-slot maps keep state attached to the request that owns it. Finishing slot 3 does not move slot 7's recurrent state, sampling history, or KV ownership. That isolation makes cancellation and immediate slot reuse safe while other users are still generating.

Paged KV and paged attention

A contiguous KV cache ties one sequence to one growing allocation. Continuous batching needs something closer to virtual memory. The PagedAttention paper (PDF) describes how fixed-size blocks and logical-to-physical mappings make KV memory reusable across requests.

Lucebox divides attention KV into fixed-size physical blocks. The block size and state layout depend on the backend. Every sequence owns a block table that maps logical token positions into the shared pool:

sequence A  [0][1][2][3] ----> blocks [7][2][9][4]
sequence B  [0][1]       ----> blocks [1][8]
free pool                ----> blocks [0][3][5][6]...

The attention operator reads the cached history by following each sequence's block table and visiting its blocks in logical token order. Packed prefill and decode can attend directly over the shared pool without gathering or relocating the cache first.

When B finishes, blocks 1 and 8 can be assigned to a new request without moving A. This makes request retirement and slot reuse cheap enough to happen continuously.

--max-concurrency selects the number of sequence slots. The continuous path requires --paged-attention. --kv-pool-tokens 0 derives physical KV capacity from available device memory, while an explicit value creates a controlled pool rounded to whole blocks.

Chunked prefill is a slot phase

Consider request C arriving with a 3,000-token prompt while A and B are already decoding. Processing all of C at once would give good prompt throughput and terrible inter-token latency for A and B.

On a backend that supports chunked prefill, the planner can select a bounded slice of C on each step. For example:

step 1  C[0:512]        + decode(A, B)
step 2  C[512:1024]     + decode(A, B)
step 3  C[1024:1536]    + decode(A, B)
...
final   finish C prompt + decode(A, B)
next    decode(A, B, C)

C already owns a stable slot. Each selected chunk appends directly to that slot's paged KV and advances its recurrent state. The scheduler keeps the slot in the prefill phase while construction is incomplete, so it cannot appear in a decode plan early. After the last slice, commit_prefill() changes the slot to the decode phase.

A chunk budget limits how much new prompt work enters a step. It does not guarantee a fixed inter-token latency: the cost still depends on the model, context length, and active work.

Packing unequal prompt chunks together

Chunking alone does not make the GPU work efficient. A scheduler could choose three fair prompt slices and still execute four separate forwards:

C: 512 tokens -> forward
D: 512 tokens -> forward
E: 180 tokens -> forward
decode rows    -> forward

A backend that supports ragged prefill can turn that plan into one token axis:

[ prompt C x512 ][ prompt D x512 ][ prompt E x180 ][ decode A ][ decode B ]
<---------------- one model traversal ---------------->

E is not padded to 512 tokens. Dense projections, normalization, feed-forward layers, attention lowering, and output work operate over the packed width. Segment metadata tells sequence-dependent operations where each prompt ends and which slot owns its state.

Shared projections run over the packed tensor while stateful operations advance each ragged segment against the correct slot. In full-attention layers, per-query causal positions and block-table metadata let prompt and decode rows coexist in the same graph. The output projection, or LM head, runs on prompt tails and decode rows. Speculative verification also needs output scores for its proposed tokens, because those scores determine which proposals the target accepts.

The scheduler limits how much prompt work enters a step. Ragged packing lets the selected chunks share projections instead of running separate forwards.

Admission and step planning are different decisions

The scheduler cannot admit a request merely because live_requests < max_concurrency.

For the normal paged-KV path, admission needs two resources:

admit(request) = slot available
              + KV capacity for prompt and rolling decode headroom

The engine reserves that capacity atomically, so a request does not occupy a slot while lacking the memory needed to finish prefill. Prefill width and token budgets are checked later, while the scheduler builds each StepPlan. They decide which already-admitted prompts advance in that iteration.

The deferred queue is FIFO. If request D at the front needs 100 blocks and only 15 are free, a later request E needing 10 blocks does not jump ahead. Strict FIFO prevents a stream of small prompts from starving D, at the cost of possible head-of-line blocking. Requests wait in arrival order until enough capacity is available, even when a smaller request could fit sooner.

DeepSeek V4 Flash on Strix Halo and two GPUs

PR #598 adds paged continuous serving for DeepSeek V4 Flash. The measurements here use autoregressive generation, without DSpark. One configuration keeps the model on Strix Halo; the other splits expert work between an R9700 and Strix Halo.

In the R9700+Strix Halo configuration, the R9700 owns dense work and frequently used experts, while Strix Halo handles the remaining experts. Every layer has to bring those results together. A second GPU adds compute capacity, but also transfers and synchronization; it does not guarantee higher throughput.

Concurrent throughput and first-token latency

C is the number of concurrent clients. Each width uses three synchronized waves of short coding prompts, repeated across the active clients, with exactly 96 output tokens per request. Both paths use the same model, with ROCmFP2 gate/up experts, ROCmFP3 down experts, and Q4_0 KV. Output-window throughput counts generated tokens from the first output until the last completion. Time to first token, or TTFT, measures the initial wait. These fixed-concurrency runs measure scaling; they do not simulate random arrivals or a mix of prompt lengths.

Throughput cells show median [minimum–maximum] across three repetitions. TTFT is the median of the per-run p95 values; these small request sets do not establish production tail latency. Both configurations reserve six sequence slots and use a 24,576-token KV pool. Measured on 2026-09-07 with PR 598 at a085cc22. Each repetition starts a fresh server and measures C1 through C4 in order.

The R9700+Strix Halo configuration enables the qualified copy-batching and grouped-expert graph settings. Batching peer copies reduces transfer overhead, while grouping expert work reduces the number of GPU launches. The graph setting reduced C4 launches from 17,168 to 8,998 in its separate qualification run. These measurements do not use the additional packed and sorted expert kernels.

On Strix Halo the gathered graph packs up to four chronological prompt rows per sequence into each step, with sixteen rows per step in total. A lone prompt can use all sixteen rows when no requests are decoding. Each row reads the preceding rows from its own sequence; dense projections share weight reads across rows, and routed experts run in groups of up to eight rows. The runtime also batches the compressor projections while keeping each sequence’s compressor state updates in chronological order. The R9700+Strix Halo configuration still advances one prompt token per lane per step. The chart and table use this updated runtime.

env ROCR_VISIBLE_DEVICES=1 \
  DFLASH_DS4_TP_SPLIT_COUNT=1 \
  DFLASH_MMID_GROUPED=0 \
  DFLASH_CUDA_MMVQ_MOE_ROWS_PER_BLOCK=2 \
  DFLASH_CUDA_MMVQ_MOE_SPARSE_WARP_BLOCKS=0 \
  DFLASH_CUDA_MMVQ_MOE_Q2_WARP_GROUPS=0 \
  DFLASH_CUDA_MMVQ_MOE_Q4_WARP_GROUPS=0 \
  DFLASH_CUDA_MMVQ_MOE_FP2_PACKED32=0 \
  ./server/build-hip/dflash_server \
  /path/to/DeepSeek-V4-Flash-ROCMFP2-STRIX.gguf \
  --host 127.0.0.1 \
  --port 18081 \
  --model-name dflash \
  --target-device hip:0 \
  --paged-attention \
  --max-concurrency 6 \
  --kv-pool-tokens 24576 \
  --max-ctx 4096 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --prefix-cache-slots 0 \
  --prefill-cache-slots 0 \
  --disk-prefix-cache off \
  --ds4-prefill exact

env HIP_VISIBLE_DEVICES=0,1 \
  ROCR_VISIBLE_DEVICES=0,1 \
  DFLASH_DS4_MOE_TP=1 \
  DFLASH_DS4_MOE_TP_INPROC=1 \
  DFLASH_DS4_MOE_TP_GPU=1 \
  DFLASH_EXPERT_BUDGET_MB=11700 \
  DFLASH_MMID_GROUPED=0 \
  DFLASH_CUDA_MMVQ_MOE_ROWS_PER_BLOCK=2 \
  DFLASH_CUDA_MMVQ_MOE_SPARSE_WARP_BLOCKS=0 \
  DFLASH_CUDA_MMVQ_MOE_Q2_WARP_GROUPS=0 \
  DFLASH_CUDA_MMVQ_MOE_Q4_WARP_GROUPS=0 \
  DFLASH_CUDA_MMVQ_MOE_FP2_PACKED32=0 \
  DFLASH_DS4_TP_BATCH_SPLIT_COPIES=1 \
  DFLASH_DS4_TP_GROUPED_MMVQ=1 \
  DFLASH_DS4_TP_SPLIT_COUNT=1 \
  ./server/build-hip/dflash_server \
  /path/to/DeepSeek-V4-Flash-ROCMFP2-STRIX.gguf \
  --host 127.0.0.1 \
  --port 18081 \
  --model-name dflash \
  --target-device hip:0 \
  --peer-access \
  --paged-attention \
  --max-concurrency 6 \
  --kv-pool-tokens 24576 \
  --max-ctx 4096 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --prefix-cache-slots 0 \
  --prefill-cache-slots 0 \
  --disk-prefix-cache off \
  --ds4-prefill exact

Qwen3.8-27B DFlash2 on the R9700 (vs llama.cpp)

PR #642 adds batched DFlash2 decode for Qwen3.8-27B concurrent serving. Qwen uses a draft model to propose several tokens, then asks the target model to verify them together. Batching gives both stages work from several requests. PR #707 also combines verification operations and removes intermediate state writes that the verification path never reads.

The Lucebox measurements use the qualified f1521495 revision on an AMD Radeon AI PRO R9700 with ROCm 7.2.4. The target is Qwen3.8-27B IQ4_XS, with a Q8_0 DFlash2 draft, Q8_0 KV, full GPU offload, and greedy sampling. Prompt, prefix, prefill, and disk caches are disabled. A benchmark-only minimum-token floor suppresses early EOS; the per-request caps select 256 or 128 output tokens. The experimental smaller MMQ tile is not enabled.

HumanEval concurrency, C1 through C5

Each concurrency runs ten synchronized waves with exactly 256 output tokens per request. Lucebox values are medians of three repetitions with a server restart before each, each preceded by a warmup wave. The llama.cpp comparison uses the same target and equivalent draft quantization. The two engines were benchmarked in separate runs.

Both throughput columns report generated tokens per second. The Lucebox server has six slots, a 16,384-token per-sequence limit, and a 98,304-token KV pool.

At C1, Lucebox produces 106.6 total tok/s. At C5, it produces 300.9 total tok/s across five requests. Median TTFT changes from 0.24 seconds at C1 to 0.68 seconds at C5. These rates describe the whole server, not each client; the total and output-window columns should be compared separately.

Throughput and latency with long prompts

The retrieval workload places an access label near the beginning of a document and asks for it at the end. Each request generates exactly 128 tokens. We record both total throughput and time to first token, because a fast output phase can coexist with a long initial wait.

The Lucebox retrieval fixtures are deterministic and archived with the measurements. The first chart compares Lucebox and llama.cpp at each prompt length with one and five clients; the second shows how long Lucebox clients wait for the first token. The full table sits under each chart.

Fixed-concurrency waves make scaling reproducible. They do not measure production queueing under random arrivals, cancellation, mixed prompt lengths, or a latency service-level target. The retrieval check is a narrow correctness screen, not a general model-quality score.

All 225 retrieval responses contain the expected label and reach the requested token count, although some responses differ between runs. The Qwen benchmark settings below give the details. At the longest prompt and C5, median TTFT is 100.0 seconds, while total throughput is 6.2 tok/s. Output-window throughput excludes the wait before the first token.

env ROCR_VISIBLE_DEVICES=0 \
  DFLASH_MIN_TOKENS=256 \
  DFLASH_PREFILL_FIRST_BURST_STEPS=0 \
  DFLASH_IDLE_PREFILL_TOKENS=4096 \
  DFLASH_MAX_CONCURRENT_PREFILLS=8 \
  ./server/build-hip/dflash_server \
  /path/to/Qwen3.8-27B-PR625-IQ4_XS.gguf \
  --draft /path/to/dflash2-q8_0.gguf \
  --draft-device hip:0 \
  --target-device hip:0 \
  --paged-attention \
  --max-concurrency 6 \
  --kv-pool-tokens 98304 \
  --max-ctx 16384 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --fa-window 0 \
  --prefix-cache-slots 0 \
  --prefill-cache-slots 0 \
  --disk-prefix-cache off \
  --admission-coalesce-ms 50 \
  --chat-template-file ./harness/benchmarks/concurrency/raw_prompt_identity.jinja \
  --host 127.0.0.1 \
  --port 18142 \
  --model-name qwen38

From one request to a shared local server

Lucebox can process several requests together and admit new ones as sequences finish. Stable slots preserve each request's state, paged KV makes memory reusable, and the backend batches the selected work. Qwen adds chunked, ragged prefill and speculative verification; DS4 uses the same scheduler for autoregressive generation.

With short prompts, Qwen grows from 106.6 total tok/s at C1 to 300.9 at C5 on the R9700. DS4 on Strix Halo grows from 23.8 at C1 to 48.4 output-window tok/s at C4. Those are different metrics and workloads, but both show one machine serving more concurrent work. The latency measurements show how long clients wait as concurrency increases.

Batching diagram explanation adapted from the Modular LLM Inference Handbook. Lucebox implementation: PR #555, paged-attention foundation; PR #594, concurrent paged serving; PR #595, packed ragged prefill; PR #598, DeepSeek V4 continuous serving; PR #659, Qwen3.8 DFlash2 batching; and PR #707, Qwen verification fusion.

Related

• Qwen3.8-27B on the AMD R9700: up to 227 tok/s
• Tool prefix caching: 48× faster warm prefill for agent loops
• Lucebox beats DGX Spark by 3.63× on DeepSeek V4 Flash decode