August 2026

By Davide Ciffa

Qwen3.8-27B on the AMD R9700: up to 227 tok/s

The lucebox engine now serves Qwen3.8-27B on a single AMD Radeon AI PRO R9700 (32 GB, RDNA4) with the DFlash2 block-diffusion drafter from z-lab: up to 227 tok/s on code, 208 tok/s HumanEval average, math at 133 tok/s, on an off-the-shelf quant that matches an 8-bit reference on HumanEval and GSM8K. And it is lossless: greedy verification commits only tokens the model itself would produce, so the drafter changes how fast you get the answer, never what the answer is.

TL;DR

• The served file is Unsloth's UD-IQ4_XS, straight off HuggingFace. Measured against a Q8_0 reference it is 8-bit-class: mean KL divergence 0.018, 94% top-1 agreement, and it matches the reference on HumanEval and GSM8K within noise. No requantization step, no custom file.
• With speculation: 208 tok/s HumanEval average, 228 peak, 133 tok/s on math, 156 tok/s end-to-end with prefill and HTTP included.
• The lever is the draft block width. DFlash2 ships trained at 8 tokens per block, but its acceptance extrapolates: --draft-block-size 16 takes the HumanEval average from 133.9 to 208.1 tok/s, and on this file widening did not change a byte of output at any width, verified below.
• Faster than llama.cpp running the same DFlash2 drafter: 3.8x on decode (208.1 vs 54.6) and 3.3x end-to-end (156.2 vs 47.2), on the same card with the same weights file.
• Plain decode: 32.3 tok/s against llama.cpp's 29.7 on the same file. This quant streams only 13.3 GiB per token but runs at about 71% of the card's bandwidth, because dynamic quants push some tensors below 4-bit onto slower kernels. A pure IQ4_XS requant reaches 37.4 tok/s plain if autoregressive speed is what you need, hosted ready-made on our HuggingFace; the quantization section below measures that trade both ways.

A block-diffusion drafter, verified exactly

DFlash2 is a 5-layer draft model that reads the target's hidden states and proposes a whole block of tokens in one masked, parallel pass, then a lightweight selector traces one coherent path through the top candidates per position. lucebox verifies the proposed chain in a single batched target forward and commits the accepted prefix. The verification step reads the full weights once whether it checks 8 or 16 positions, which is why wider blocks are nearly free, and why one R9700 can turn 32 tok/s of physics into 200+ tok/s of code.

The engine work behind this release: the DFlash2 dynamic convolutions and candidate selector in the draft graph, a context-KV ring cache so drafting stays flat at any context, a fused DeltaNet decode path for Qwen3.8's hybrid linear attention, RDNA4-tuned matmul tiles, and the widened draft horizon with byte-exactness validated at every width. Full details and A/B history are in PR #625.

What the block width buys, measured as decode-only speed (the generation phase alone, prefill excluded, as reported by the engine's own per-request timer):

All rows: Qwen3.8-27B UD-IQ4_XS, greedy, single R9700, ROCm 7.2. The 208.1 figure is the mean across the ten prompts, weighted by tokens generated; 227.8 is the fastest individual request, on the kind of highly predictable code where the drafter commits its full block. Expect the average, not the peak, on mixed work.

Against llama.cpp, with the same drafter

Comparing a speculative engine against plain decoding is not a fair fight, so we gave llama.cpp the same weapon. Upstream now supports DFlash2 through PR #27342 (--spec-type draft-dflash). We built that branch for RDNA4 and ran it on the same card, the same UD-IQ4_XS weights file, the same z-lab drafter checkpoint, and the same ten HumanEval prompts at 256 tokens through the OpenAI chat endpoint.

Read the columns for what they are. Decode is the generation phase alone, where the engines actually differ: 208.1 against 54.6, a 3.8x gap on the identical drafter and identical weights. End-to-end is the whole request including prefill and HTTP, which is what a user feels: 156.2 against 47.2, or 3.3x. Prompt processing is not the story on these short prompts; the long-context section below measures prefill at length for our engine, and decode at length for both.

How these numbers were taken

Both engines were driven by the same client script against the same OpenAI-compatible endpoint, ten HumanEval-style code prompts, max_tokens 256, temperature 0, one request at a time. The end-to-end figure is total tokens divided by total wall time across all ten requests, so prefill, sampling and HTTP overhead are all inside the number, and both servers were freshly started for their run so both carry the same first-request warmup cost. llama.cpp is PR #27342 built for gfx1201 with -ngl 99 -fa 1 -c 8192 -ctk q8_0 -ctv q8_0, drafting with --spec-draft-n-max 7. Before anyone asks whether the width flag is the trick: llama.cpp clamps that flag to the drafter's trained block, 8 for this checkpoint, and we measured 7 and 16 (clamped to 8) within noise of each other, 54.6 against 54.3 decode. Drafting past the trained block is exactly the part that takes engine work, and it is worth 133.9 to 208.1 tok/s on our side. The two engines use different drafter file layouts, so llama.cpp ran the official checkpoint from incoai/Qwen3.8-27B-DFlash2-GGUF and lucebox ran the same z-lab weights through our converter.

Decode-only figures come from each engine's own per-request timer (lucebox reports decode_tokens_per_sec, llama.cpp reports predicted_per_second), so neither side is measured by our stopwatch. For reference, llama.cpp's standalone llama-bench gives 30.9 tg128 and 1,283 pp512 on this file, consistent with the server numbers above.

Two different lucebox figures appear in this post and they measure different things: 156.2 is end-to-end including prefill, which is what the comparison above needs; 208.1 is decode-only, which is what the block-width table reports. Neither is a marketing number for the other.

What "lossless" means here, precisely

The drafter never puts a token into the output. Every token is checked against the target model's own logits and only committed if the target agrees, so acceptance rate changes speed and nothing else. Measured on the served file: blocks of 8, 12 and 16 produce hash-identical bytes on all ten prompts, and all ten are also byte-identical to plain decode.

There is one honest subtlety worth stating plainly, because anyone who tests this on another file may find it. Verification evaluates sixteen positions in one batched matrix multiply, while plain decode evaluates one position at a time through a vector kernel. Those two paths sum floating-point numbers in a different order, so when two candidate tokens sit within floating-point noise of each other the winner can differ. On the served file all ten prompts matched plain decode exactly; on the pure requant measured in the quantization section below, one of ten diverged after 536 identical characters, where plain decode wrote return result and the speculative path wrote an extra if current: result.append(current) before it. Both are the model's own argmax under their respective arithmetic. This is a property of batched evaluation in general, not of speculation specifically: changing batch size in any engine can move a tie. What speculation does not do is substitute the drafter's opinion for the model's.

Against llama.cpp running the identical drafter on the identical file, lucebox is 3.3x faster end-to-end. The gap is not the model, it is what the engine does with it, and it is not the width flag either: their integration is complete, with target-feature extraction and the DFlash2 selector both implemented, but their selector samples candidate paths with an RNG where ours runs a deterministic best-first search over normalized scores, and ours keeps the drafter's features exactly current across rollbacks. Acceptance is where those details compound: lucebox commits long runs of its 16-token block on predictable code, a 6.4x gain over its own plain decode, while llama.cpp gains 1.8x over its own at either width setting. One target forward reads 13.3 GiB of weights whether it checks 4 positions or 16, so the engine that fills that batch wins.

What happens at long context

Short benchmarks flatter speculative engines, so here is the same stack on real long prompts: a growing slice of this repository's C++ source, with a question at the end that requires both reading a label planted at the very top and summarizing the code. Generation is 128 tokens, greedy, full attention, block 16, --max-ctx 131072.

All of that is on one 32 GB card with no offload and no paging. The KV cache at q8_0 costs about 34 KiB per token, so 128K tokens is 4.2 GiB on top of 13.3 GiB of weights, which is why --max-ctx 131072 fits comfortably; even 256K would fit at roughly 22 GiB total. The label planted at the very top of the file was recalled at every length up to 70K, so this is real context, not a truncated window.

Two things are worth being blunt about. First, decode here is 24 to 58 tok/s, not 208. That is mostly a workload difference, not a long-context penalty: this task answers a question in prose, where the drafter's block acceptance sits between 0.21 and 0.36, while HumanEval-style code completion runs its decode 6.4x over plain. Speculation pays exactly in proportion to how predictable the next tokens are, and any engine that quotes one number for every workload is quoting the friendly one.

Second, the slope past the first rows is set by the attention kernel, not by weight traffic: its cost grows with KV length and with the number of query rows it checks, so a wide draft block is a large win on short prompts and a growing tax on long ones. The engine therefore caps the verify batch at 8 rows once the context passes 8K, on context length alone and never on the observed acceptance rate; the drafter still proposes its full block, only the verified width narrows. Short prompts are untouched, and the narrowing is reproducible for a given prompt rather than chasing a moving target.

The comparison narrows at length. Running llama.cpp with the same DFlash2 drafter over the same growing prompts, decode on its own timer:

From 1.5K through 36K the two engines are within a few percent of each other, inside run-to-run spread. Past that they pull ahead: at 70K they hold 31.1 against our 23.7, so on unpredictable text at extreme context their step time degrades less than ours does. We would rather print that row than leave it out; it is the next thing to fix, and it is not draft width, because both engines are checking the same number of positions there. Our large margin is on code and math at the prompt lengths where people actually iterate, which is what the model is for.

To make that concrete on the same files, we ran a code-continuation ladder over the identical context lengths: lucebox holds 142 down to 45 tok/s from 1.4K to 61K prompt tokens, while llama.cpp with the same drafter sits between 28 and 54 and passes us only once, at 42K. The chart puts both workloads side by side; the long-context story depends entirely on whether the drafter can predict what comes next.

Both engines run the same weights file and the same drafter. Solid lines: code continuation, the workload speculation is built for. Dashed: prose summarization, the drafter-hostile case, where both engines converge toward plain-decode speed. The lucebox code line is jagged because acceptance moves with the content of each slice (0.48 to 0.84 per block).

About the quantization

The served file is Unsloth's UD-IQ4_XS, used exactly as downloaded. That choice came out of measurement, not convenience, and the measurements are worth showing because two other candidates nearly won: bartowski's stock IQ4_XS, and a --pure requant of it we made ourselves, which is the fastest file this card has served. Quality first, against a Q8_0 reference of the same model. The reference is 27.12 GiB and does not fit next to the drafter in this card's 32 GB; the second GPU in this lucebox, a Strix Halo with 122 GB of usable unified memory, ran it as the offline quality anchor.

How these were measured. Perplexity and KL divergence run over 300 sequential 512-token chunks of the wikitext-2 test set, about 153K tokens, with llama-perplexity --kl-divergence against saved Q8_0 logits. That comparison is paired per token, which is what lets it resolve differences the raw ±0.06 error bars cannot: the PPL ratios above carry uncertainties under a tenth of a percent. Top-1 agreement is how often the quant's argmax token matches the reference's. HumanEval is all 164 problems, generated through the same serving path as every other number in this post, greedy with the DFlash2 drafter attached (verification is exact, so the drafter cannot change what the model writes), and executed against the official tests. GSM8K is the first 200 test problems, scored on the final answer.

Speed next: same engine, same client, same ten HumanEval prompts, block 16.

Read the two tables together and the decision writes itself. The Unsloth file is 8-bit-class on every quality instrument we have, matching or beating the Q8_0 reference outright on both task evals, and it ties the stock file on speed. One detail is instructive: it is the smallest file in the table yet not the fastest, because dynamic quants push selected tensors below 4-bit and the sub-4-bit kernels on gfx1201 run far under IQ4_XS bandwidth. Bytes only buy speed when the kernels that read them keep up.

The pure requant is the opposite trade: about 12% more end-to-end speed than the served file on code, and 37.4 against 32.3 tok/s on plain decode, for a quality cost that is small but real and shows up on every instrument: the KL divergence rises by half, top-1 agreement gives up two points, and it hands back seven HumanEval and six GSM8K problems against the served file. Each task gap on its own sits within a couple of standard errors at these sample sizes, but they all point the same way, and the KL divergence, the sensitive instrument here, is unambiguous. An earlier version of this post served that file and called its perplexity gap unresolvable inside the error bars; the paired comparison resolves it, and we would rather correct ourselves than round in our own favor. We now serve the file whose quality we can defend without a footnote, and the 12% is yours if you want it: we host the ready-made file at Lucebox/Qwen3.8-27B-IQ4_XS-pure-GGUF with the trade stated on the card, so there is nothing to requantize. Or reproduce it yourself with one command:

# from bartowski/Qwen3.8-27B-GGUF
llama-quantize --allow-requantize --pure \
  models/Qwen3.8-27B-IQ4_XS.gguf \
  models/Qwen3.8-27B-IQ4_XS-pure.gguf IQ4_XS

Run it

# build (ROCm 7.2+, RDNA4)
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
cmake -S server -B server/build-hip -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DDFLASH27B_GPU_BACKEND=hip \
  -DDFLASH27B_HIP_ARCHITECTURES=gfx1201 \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DGGML_HIP_NO_VMM=ON
cmake --build server/build-hip --target dflash_server -j"$(nproc)"

# the target quant, used as downloaded, and the z-lab drafter converted to GGUF
mkdir -p models
huggingface-cli download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-UD-IQ4_XS.gguf --local-dir models
huggingface-cli download incoai/Qwen3.8-27B-DFlash2 --local-dir models/dflash2
python server/scripts/convert_dflash_to_gguf.py \
  models/dflash2/model.safetensors models/qwen38-dflash2-f16.gguf
python server/scripts/quantize_dflash_draft.py \
  models/qwen38-dflash2-f16.gguf models/qwen38-dflash2-q8_0.gguf --scheme q8_0

# launch: the configuration every number in this post was measured with
# (device auto-detects; on a multi-GPU box add --target-device hip:N)
./server/build-hip/dflash_server models/Qwen3.8-27B-UD-IQ4_XS.gguf \
  --draft models/qwen38-dflash2-q8_0.gguf \
  --draft-block-size 16 --max-ctx 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --port 8216

# it speaks the OpenAI API, so point any client at it
curl -s http://127.0.0.1:8216/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Write a Python LRU cache."}],
       "max_tokens":256,"temperature":0}'

There is nothing else to set. Earlier builds needed four environment variables to reach these numbers, which was a bad default: the tuned path is the one everyone should get, and requiring opt-in mostly means running the slow path by accident. Exact f32 chain checkpoints, rollback from the first accepted token, memoized q8 activations and skipping the precision-neutral KV rotation are all defaults now, so the command above is the fast one. We also checked the obvious KV knob: --cache-type-k f16 --cache-type-v f16 doubles KV memory and measured slightly slower on this file, 202.5 against 208.1 decode on the same ten prompts, so the q8_0 cache in the command stands.

--draft-block-size 16 widens the drafter to double its published horizon; the server accepts up to 2x the checkpoint metadata and startup logs confirm the exact-verify path. On prose-heavy workloads the drafter accepts less and the widening stops paying, so leave the flag off there; code and math workloads are where it shines. The target file is served exactly as downloaded; the quantization section above measures the alternatives if you want to trade differently.

Bottom line

A single R9700 now serves a 27B frontier-family model at interactive-agent speeds: 208 tok/s average on real code, peaks up to 227, and 3.3x llama.cpp running the very same drafter on the very same weights file, with every token verified against the model's own logits rather than trusted from a drafter, on a stock quant that matches an 8-bit reference on the task benchmarks we could throw at it. The whole stack is open source.

Source: github.com/Luce-Org/lucebox (PR #625). Drafter checkpoint: incoai/Qwen3.8-27B-DFlash2 (z-lab). Numbers measured on one Radeon AI PRO R9700 32 GB, ROCm 7.2, August 2026.

Related

• DFlash on ggml: up to 207 tok/s Qwen3.5-27B on a RTX 3090
• Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context
• Lucebox beats DGX Spark by 3.63x on DeepSeek V4 Flash decode