July 2026

By Davide Ciffa

Tool prefix caching: 48× faster warm prefill for agent loops

Agent turns often resend the same tool definitions. Lucebox can now restore that stable prefix and prefill only the conversation that came after it.

Tool definitions are part of the prompt. In an agent with many tools, they can be larger than the conversation itself. Our test uses 24 tools with eight parameters each. The prompt reaches about 15,000 tokens before the agent has done much work.

Most of those tokens are identical on the next turn. Lucebox now saves the model state at the end of that stable prefix. Later requests restore it and process only the new messages and tool results.

The result

We ran one cold request and three growing warm turns with Qwen3.6-27B Q4_K_M on a Ryzen AI MAX+ 395. The cold prefill took 50.35 seconds. The warm median was 1.04 seconds.

Cold time divided by the warm median is 48.33×, rounded to 48× in the title. Changing one tool description returned prefill to 51.18 seconds and produced a cache miss.

Why the old cache missed

Lucebox already had a cache for exact prompt matches. That helps when the complete prompt repeats. Agent conversations do not repeat: every tool result and every new message changes the prompt.

turn 1  | system and tools | user request | assistant start
turn 2  | system and tools | turn 1 | tool result | new request
turn 3  | system and tools | turn 1 | turn 2 | new request
          same prefix       conversation keeps growing

The old cache saw three different prompts. The new cache also looks at the stable boundary between tools and conversation. The first turn still does a normal prefill. The next turn can start from the saved state.

What the server restores

For Qwen3.5 and Qwen3.6, the snapshot contains more than attention KV. The backend also needs recurrent state, convolution state, and sometimes the final token seed. Lucebox saves them together so every piece comes from the same point in the prompt.

A hit requires the exact same rendered token prefix. Change a tool name, description, parameter, system prompt, or chat template and the server starts cold. Failed or cancelled requests never publish their reserved snapshot.

KVFlash solves a separate problem. It controls how much long context attention KV stays on the GPU during generation. Tool prefix caching avoids repeated prefill work between requests.

How we measured it

All three warm turns returned identical nonempty output. The telemetry also confirms that each request restored the expected prefix and evaluated the expected suffix.

A real Hermes loop

We then ran Hermes Agent 0.19.0 through the public Lucebox harness on the same machine and model. Hermes received its normal 18 tool definitions and executed three terminal commands in sequence. The initial request contained 15,615 tokens. We used a 65,536 token context because Hermes requires at least 64K, and set temperature to zero so the two runs followed the same path.

Both runs sent the same four prompt sizes, executed the same commands, and returned the same correct values. The first request remained cold. Each later request restored 15,360 tokens and evaluated only the growing conversation. The machine readable result includes every request.

A check against main

We also compared main with the cache branch using Qwen3.5-9B as an internal test model. Cold prefill was nearly equal at 16.84 and 16.97 seconds. Median warm prefill was 17.20 seconds on main and 0.37 seconds with the cache. Because this model is not an advertised Lucebox target, we use it only as a supporting check.

Checks and limits

The supported model run checks nine conditions. It covers the first miss, three later hits, exact token counts, identical nonempty output, and a miss after changing one tool. All nine pass. The server suite also passes 2,091 assertions with zero failures.

The 48× result is one validation sequence, not a distribution across agent workloads. The Hermes result is also one controlled task. We did not measure other tool sizes, memory used by each snapshot, or contention between concurrent clients. A longer test covers four slot wraparound, but sustained multiagent load is outside the scope of this result.

Using it

Clients keep sending normal OpenAI compatible chat completion requests with tools attached. There is no cache flag in the request and no second server to run.

The native HTTP server keeps 32 prefix cache slots by default. CUDA and ROCm containers use the same default. Set DFLASH_PREFIX_CACHE_SLOTS to choose another value, or set it to 0 to turn the cache off. The benchmark used four slots:

dflash_server model.gguf \
  --max-ctx 32768 \
  --prefix-cache-slots 4 \
  --prefill-cache-slots 2

Each response reports cache activity in usage.timings:

"timings": {
  "cache_hit": true,
  "cached_prefix_tokens": 14848,
  "prefilled_tokens": 237,
  "effective_prompt_tokens": 15085
}

The effective token count grows with the conversation. The public benchmark script reads these fields directly instead of inferring a hit from elapsed time:

python3 server/scripts/benchmark_tool_prefix_cache.py \
  --url http://127.0.0.1:8080 \
  --json-out /tmp/tool-prefix-cache.json

Implementation

David Roth opened PR #492 with the first prototype. The merged implementation lives in the production C++ HTTP path. It no longer needs the separate Python server or the earlier SNAPSHOT_THIN and RESTORE_CHAIN protocol.

The cache works alongside PFlash, FlowKV, and KVFlash. The system prompt, tools, and recent turns stay verbatim. Older messages can still be compressed, while KVFlash manages the live attention footprint during long context generation.

Source: Lucebox PR #492, merged as c3b71e4b on July 27, 2026. Benchmark figures and validation details come from the merged PR and its included benchmark script.

Related

• Launch and tune Lucebox with real agent harnesses
• PFlash: 10× prefill speedup over llama.cpp at 128K on a RTX 3090
• Luce KVFlash: 256K context with 72 MiB of KV on the GPU
• Read and review PR #492 on GitHub