Engine / open source
The engine that makes big models run on one workstation.
Lucebox ships with our open source inference engine: speculative decoding and prefill, custom CUDA and HIP kernels, and a scheduler that can put one model across two GPUs or keep two models resident at once. Every figure below is measured, with the report or the code behind it.
01How the engine works
A small model proposes, the real model checks.
A draft model proposes several tokens at once. The full model verifies them in one batched pass and keeps the ones it agrees with. Verification reads the weights once whether it checks four positions or sixteen, which is where the speed comes from.
- Lossless
- Greedy verification commits only tokens the model itself would produce
- Decode
- DFlash and DFlash2 draft blocks of tokens, checked against the target logits
- Prefill
- PFlash proposes the prompt pass, so long prompts do not wait on one sequential read
- Memory
- KVFlash pages cold KV chunks to host RAM, bit-exact, to keep long context on one card
- Kernels
- Hand-tuned CUDA and HIP kernels per architecture, plus a fused megakernel path
- Licence
- Apache 2.0, C++17, CUDA 12+ and HIP 7+
02Models available today
The models the engine serves, with the speedup measured on each.
Each row is a published model file with the drafter it runs with. The speedup is what the engine adds over plain decoding on the same hardware.
- Qwen3.8-27B
- UD-IQ4_XS with the DFlash2 drafter: 6.4× over plain decode, 208.1 tok/s HumanEval average on one R9700
- DeepSeek V4 Flash
- 284B, ROCmFPX build with the DSpark drafter: 51.1 tok/s decode across two AMD GPUs
- Laguna XS 2.1
- 33B from poolside, with PFlash and KVFlash: 6.1× prefill at 256K, 411 s down to 67.3 s
- Gemma 4 31B
- Q4_K_M with a DFlash drafter: 3.2× decode
- Gemma 4 26B-A4B
- Q4_K_M with a DFlash drafter: 1.31× decode
- Your own weights
- Any GGUF you bring; the CLI loads it and the drafters are optional
03Two GPUs, one model
The workstation runs one model across both processors.
A 284B model does not fit in 32 GB of VRAM. The engine gives the two AMD GPUs different jobs instead of splitting every operation evenly, so each one does what its hardware is good at.
- R9700
- Dense path, the experts picked most often, the draft model, the working cache and token selection
- Strix Halo
- The remaining experts in 128 GB of unified memory, run in parallel for the current token
- Measured
- 86 tok/s decode and 788 tok/s prefill at 2K on DeepSeek V4 Flash
- Against one DGX Spark
- 3.63× the decode speed, on the same 284B model
04Two models at once
Keep one model hot and a second one ready.
128 GB of unified memory next to 32 GB of VRAM means a second model does not have to wait on a download. One model stays resident on the GPU while another sits in unified memory, ready to serve.
- Hot model
- Resident in the 32 GB of VRAM, serving the latency-sensitive path
- Second model
- Held in the 128 GB of unified memory, ready to swap in
- Routing
- The CLI loads the profiles and routes each request to the right model
- Placement
- Model placement and load balancing are separate controls in the workstation
05Workloads
What the engine is tuned for, and what it costs elsewhere.
Speculation pays in proportion to how predictable the next tokens are. Code and agent loops gain the most, prose gains the least, and the reports say so with numbers rather than one headline.
Order
The workstation this engine was written for, from $5,999.
The engine is open source and runs on many cards. Lucebox ships it installed, with the qualified model profiles already tuned for the hardware inside.