By Davide Ciffa

Lucebox cluster: DeepSeek V4.1 Flash at up to 103 tok/s on two boxes and one 100 GbE NIC link

Two Luceboxes joined by one 100 GbE cable run DeepSeek V4.1 Flash as one model, at up to 103 tok/s writing code. That is 1.9 times one Lucebox Zero 495 with 192 GB running the same engine. Each step sends about 95 small messages between the boxes, one after another, so what counts is how fast each one crosses.

TL;DR

• Two boxes, one model. Two Luceboxes share DeepSeek V4.1 Flash over a direct 100 GbE RDMA link. Each is a Ryzen AI MAX+ 395 with 128 GB and a Radeon AI PRO R9700. Each holds half of the experts, all of them in memory.
• Nearly twice the biggest single box. Up to 103 tok/s writing code, against 54 on one Lucebox Zero 495 with 192 GB running the same engine. Prose after a 2K-token prompt gains less: 56 against 37.
• Every message counts. A step sends about 95 messages of 40 to 100 KB, one after another, so the time each one takes to cross adds up. RDMA on the ConnectX writes each one straight into the other box's memory, with no network stack in between.
• A connection built for small messages. RDMA writes straight into pinned memory, bf16 on the wire and kernels that raise their own completion flag. The integrated GPU handles the network side.
• From 60 to 103 tok/s. The first two-box run was only a quarter faster than one box. Each step after that removed a wait between the boxes, a trip through the host or a kernel.

Below, both setups answer the same code prompt, replayed from the timings we recorded. Both clocks start at the first token, and each side shows the answer it actually returned.

Two boxes, one cable

Each Lucebox here is the Ryzen AI MAX+ 395 build: 128 GB of unified memory and a 32 GB Radeon AI PRO R9700. On its own, a 128 GB box can't keep all of DeepSeek V4.1 Flash's experts in memory. In our DeepSeek V4.1 Flash post it read the rarely used ones from the SSD and wrote code at up to 26 tok/s.

Two boxes have room for all of them. A ConnectX-5 Ex card in each box joins them with one 100 GbE cable, with no switch in between. The two engines talk over RDMA (RoCE v2) and run as one model: every token goes through every layer on both boxes at once.

Split this way, the boxes add up their halves twice in every layer, after attention and after the experts. Nothing comes from the SSD while the model runs. As on one box, the placement is fitted to the pair, and the router picks the expert in fast memory when two choices are close.

Why a 100 GbE ConnectX

A decode step checks the drafter's next few tokens in one pass through all 40 layers. With the drafter and the output head, that makes about 95 messages a step, each 40 to 100 KB, one after another. Every microsecond a message takes is paid about 95 times a step.

Two things set how fast a message crosses. RDMA lets the card write it straight into the other box's memory, with no network stack and no extra copies on the way. The link rate sets how long it spends on the wire: 4 µs for 50 KB at 100 Gbit/s, 16 µs at 25 Gbit/s. Saving 12 µs on each of 95 messages is more than 1 ms a step. Reading a long prompt moves far more data than writing tokens, and there the link rate counts even more. That is why each box has a 100 GbE ConnectX.

The rest of a message's time goes to getting it out of one box's memory and into the other's: one RDMA write of 51 KB took 35 µs end to end.

Inside the engine, a typical message waits 18 µs for the other box's half, and a few wait much longer. Over about 95 messages a step, waiting adds up to roughly a tenth of the step.

Building the connection

Five choices made the difference:

• RDMA writes, no collective library. ROCm 7.2's RCCL ships no kernels for the Ryzen AI MAX+ GPU. Each box writes its half straight into the other box's pinned memory, and a small thread on each side passes the completion flags.
• Half the bytes. Partial sums cross in bf16.
• Kernels raise their own flag. The kernel that packs a message also marks it ready, with one memory fence instead of a separate stream operation. Sending a message and seeing it acknowledged went from 16 to 18 µs down to 9 to 11 µs on the R9700.
• The integrated GPU does the network side. The R9700 reaches the rest of the box over PCIe, so every message it sends or reads crosses that link. The integrated GPU shares memory with the CPU and the NIC's buffers. So it adds up the expert outputs of both GPUs and both boxes, and the R9700 sends its part once and reads the sum once.
• Split the big readers. Checking drafted tokens and drafting new ones both read the model's 703 MB output matrix. Each box now reads half of it, and one tiny message agrees on the winning token. The drafter is split the same way.

GPUDirect RDMA, where the NIC reads and writes the R9700's memory directly, works on this hardware too. It ran at 1.1 to 1.7 GB/s, slower than going through host memory, so the engine doesn't use it.

Two boxes don't give exactly the same answer as one box. The boxes add their partial sums in bf16, and the router breaks close calls differently, since each setup has its own placement. When two tokens are nearly tied, that can flip the pick. In the replay, the answers split at line 11. Every drafted token is still checked by the full model.

It started from maikzz32's work on Strix Halo clusters, which already ran DeepSeek V4 Flash across boxes over RoCE.

From 60 to 103 tok/s

The first two-box run wrote code at 60 tok/s, about a quarter faster than one 192 GB box at the time. Most of what followed was about the two boxes waiting for each other, and then about the GPU work in between.

• Batching and fused kernels. The check runs attention for its drafted tokens together instead of one by one, and many of the drafter's small kernels merge.
• The integrated GPU on the network side. The biggest single step: the R9700 stopped crossing its PCIe link twice for every message.
• Split output head. Each box reads half of the output matrix. The second box's R9700 also stopped going idle while the first one drafted, which saved about 1 ms a step.
• Kernel flags and fewer host round trips. The kernels that pack messages publish their own completion. The drafter's context and the rows a rejected draft restores stay on the GPU instead of passing through the host.
• Split drafter. Both R9700s draft, each with half of the drafter.
• 8-bit expert inputs. The R9700 hands each layer's input to the integrated GPU already in the 8-bit form its expert kernels read. It crosses PCIe in a quarter of the bytes, and the answers stay identical.
• Fused kernels and expert prefetch. The shared expert's two input products and its activation now run as one kernel on the R9700. On the integrated GPU, each expert kernel loads the next expert's weights while it computes the current one.

Where the time goes now

A step takes about 41 ms and keeps 4.25 tokens on average when writing code. Drafting takes about 3 ms, picking the drafted tokens 1 ms and checking them 37 ms.

Inside the check, the messages cost in two places. Waiting for the other box takes about 3.6 ms. Packing each message and adding the one that comes back takes another 2.5 ms on the R9700.

The two boxes are evenly matched: each waits for the other about as long, so moving work from one box to the other would gain little.

Prose after a 2K-token prompt runs at 55 to 59 tok/s. The drafter guesses prose less well, so each step keeps 2.5 tokens instead of 4.25.

Hardware and setup

Bottom line

DeepSeek V4.1 Flash doesn't fit in one 128 GB box. On two, it writes code nearly twice as fast as on the biggest single box. Decode sends many small messages, one after another, so the link has to be fast for each one: the 100 GbE ConnectX writes every message straight into the other box's memory over RDMA, and the engine keeps each step on the GPU.

Related

• Running DeepSeek V4.1 Flash on the Lucebox memory hierarchy: up to 3x faster with the R9700
• Lucebox (AMD Radeon AI PRO R9700 + Strix Halo) Beats NVIDIA DGX Spark by 3.63x on DeepSeek V4 Flash Decode Speed
• ROCm beats Vulkan on Strix Halo