October 2026

By Umberto Breglia

PFlash v2: 3x faster processing on long prompts

If you ask questions over long documents, retrieved passages or a whole codebase on one GPU, reading the prompt can be most of the wait before the first token. PFlash sits in front of the model and decides which parts of a long prompt the model actually reads. On six public long-context sets it answers 92 of 124 questions against full prefill's 94, while the model reads a median 27% of the prompt, and on one GPU the first token arrives 3.7× sooner on average.

TL;DR

• This is a new PFlash. Every stage of the first PFlash was redesigned, and it is now measured on long-context sets that are hard in different directions, from evidence that shares no words with the question to bugs planted in 82K–98K-token repositories.
• PFlash reads the whole prompt with a small model, then keeps the parts that matter. A 0.8B drafter reads everything, a segment probe cuts the prompt into whole units (paragraphs, functions, records, turns), a request probe finds the question, a learned relevance head ranks the units against it, and a budget decides how many fit. The big model answers from what is kept.
• Quality. 92 of 124 public questions against full prefill's 94. On LongCodeQA, 59 of 114 against 57. On our private pflash-bench, 94 of 102 against 98. The largest gap is on LongBench-v2: 4 of 12 against 7.
• Speed. On one R9700 the first token comes 3.7× sooner on average, and 4.7× sooner on 82K–98K-token code prompts: an 84K-token CodeDebug prompt goes from 132.0 to 28.4 seconds.
• One mechanism for requests, chats and agent sessions. PFlash only shortens long content the server has not read yet, one message or tool output at a time, and never rewrites what the cache already holds. The model is told where lines were left out, and they come back when a later question needs them.
• Chats. On a 255-turn multi-turn benchmark a whole conversation waits about half as long as with full prefill on one R9700 (85 against 160 seconds), with 189 answers right against 197. The losses sit in later turns; how much a follow-up recalls is a setting, and we have not tuned it on any benchmark here.

Why a new PFlash

The first PFlash post showed the compressor on needle-in-a-haystack: hide a sentence in a long filler document, ask for it back, and check that compression kept it. That test is saturated: full prefill, the first PFlash and today's PFlash all score 100 of 100 on our 100-case sweep from 16K to 120K tokens. This release redesigns every stage, and the benchmarks that follow are chosen to be hard in different directions.

How PFlash works

A long prompt is mostly material the answer does not need. PFlash finds the part that matters before the big model sees any of it. A small drafter reads the whole prompt once. Two tiny probes on its activations cut the prompt into units and find the question. A relevance head ranks the units against that question. A budget decides how much to keep, and the kept units go to the target model in their original order.

When PFlash steps aside

Compressing is a bet that the kept units are enough. PFlash only makes that bet on long content the server has not read yet, and serves everything else as it is.

• Only long content that is not already cached. PFlash acts when a request brings at least 16K tokens the server has not seen. A conversation whose history sits in the prefix cache reuses it instead: throwing a cache away to compress would only be slower.
• One message at a time, never the cache. In a chat or an agent session PFlash shortens only the newest long message or tool output, cutting at line starts. Everything before it stays exactly as it was served, so the prefix cache keeps working turn after turn (see Multi-turn chat below).
• Not a cold agent history. In a coding-agent session, the record of what the agent did and what came back is the state it acts on. When the history is not in memory, it is restored from the SSD session cache or, if the disk does not hold it either, prefilled in full; only new long tool outputs and pastes are shortened (see Agent sessions below).
• Not a cold chat whose relevance is too flat. A long chat that arrives with no cache, or a new request whose long message has too few lines to cut, is compressed as one prompt. Before that, the server checks how spread out the relevance is. If it is spread over far more units than the budget can keep, as in “list every message that mentions X” over a long mailbox, the request gets full prefill instead. Inside a running conversation a flat ranking is served as it is: every gap is marked, and a later question can bring lines back.
• Not when scoring or cutting fails. If the drafter's scores are missing or invalid, or a message cannot be cut cleanly into units, it is served whole.
• An SSD session cache keeps cold starts rare. A session's state is written to disk when it leaves memory and when the server shuts down, together with what PFlash kept of its long messages, so a resumed chat or agent session restores what the model had already read instead of recomputing it.

Benchmarks

The sets are hard in different directions. Document QA is the RAG shape: a question over many retrieved passages, most of them irrelevant (its 15 RULER QA prompts are reported separately as well). LongBench-v2 is a standard long-context suite. NoLiMa hides needles that share no words with the question. CodeDebug is a repository with a planted bug, at 82K to 98K tokens. BRIGHT asks reasoning-heavy questions whose supporting passage shares little wording with them. LongCodeQA asks questions about real repositories at 32K, 64K and 128K tokens, filtered to questions that cannot be answered without the code. pflash-bench is our own private set of real-use prompts: 102 single requests and 12 chat sessions with 68 turns. Its prompts are not published, so its rows are not reproducible from outside.

Quality

PFlash matches full prefill on Document QA, RULER QA, NoLiMa and BRIGHT, gets one more CodeDebug bug and two more LongCodeQA questions, and falls behind on LongBench-v2: 4 of 12 against 7, three questions that full prefill answers and none the other way. In two of those three the drafter's relevance scores were spread almost evenly across the prompt, a sign that no small part of it holds the answer. Over the six public sets PFlash answers 4 questions that full prefill misses and misses 6 that it answers, a net of 2 fewer (95% interval −7 to +3), inside the noise at this size. On our private pflash-bench it answers 94 of 102 against 98.

Time to first token

The gain grows with the prompt: the target's prefill grows faster than linearly with length, while PFlash hands it a fraction of the prompt. On short prompts there is little to win. NoLiMa shows where PFlash starts: 13 of its 24 prompts are around 16K tokens, too short for PFlash to act, so they are served whole in the same 15.2 seconds as full prefill and set the median, while its 32K prompts drop from 35.4 to 11.4 seconds.

Multi-turn chat

A chat is where the cache rules matter most. A long document arrives in one turn, and every later turn resends it along with the conversation so far. PFlash handles a chat one message at a time and never rewrites what the cache already holds.

• Only the newest long message is shortened. When a turn brings 16K or more new tokens (documents in the first message, a paste later on), PFlash shortens that message the way it shortens a single request and leaves every earlier turn as it was served. The cached prefix stays valid, so a follow-up prefills only its own turn.
• Recall only when the new question needs it. On each new question the drafter scores the lines left out earlier against it. If they hold less than a small share of the question's relevance, a threshold we call θ (7.5%), the question is answered from what the model already has. Otherwise the best of them come back as short excerpts at the top of the new turn, labelled as coming from an earlier message, and the cache still holds everything before them.
• The drafter keeps its work. Between turns the drafter's state waits in host memory, so a follow-up scores only what is new.

We measure this on MTRAG, IBM's multi-turn RAG benchmark, in a variant where a long document arrives in the middle of the chat: 32 conversations and 255 turns, with about 44K tokens of documents in the first turn and another 44K pasted at the third. The model's own answers are fed back as the conversation, as a chat client would do; greedy decoding, thinking off, up to 1,024 output tokens, prefix cache on. The prefix cache keeps each turn up to its last message, so with full prefill a follow-up reads only what is new. MTRAG's reference answers are free-form, so a judge (Codex Sol 6.1, blind to the arm) compares each answer with the reference.

PFlash wins on the long turns: the first answer comes in 14.7 seconds instead of 53.1, and the answer to the pasted document in 25.9 instead of 93.7. Follow-ups cost more, 5.4 against 2.5 seconds, because PFlash scores each new question on the drafter and prefills what recall brings back, yet over a whole chat it comes out well ahead, a median 85 seconds of waiting against 160. Quality is close but not level: 189 of 255 turns right against 197, a difference inside the noise at this size (95% interval −18 to +2 turns), with PFlash ahead over the first four turns and behind from the fifth on, where late questions reach further back. How much a follow-up recalls is set by θ, chosen once on a development replay outside every benchmark here; a lower value recalls more and may answer better on long chats, at some cost in follow-up time.

Agent sessions

An agent session follows the same rule as a chat: PFlash shortens only new long content, and everything the server has already read is reused as it is.

• A new long tool output or paste gets PFlash. When a step brings 16K or more tokens that no cache holds (a large file read, a long log, a pasted manual), PFlash shortens that one message, cutting at line starts. The shortened result opens with a note saying how many of its lines are shown and that the agent can call the tool again for an omitted part, and a later question from the user can bring omitted lines back through the same θ gate as a chat.
• Everything else is served whole. The earlier conversation comes from the prefix cache, at no prefill cost and with nothing left out, so compressing it could never be faster or better. New content under 16K is small enough to prefill whole.
• The history is never compressed, even cold. The record of what the agent did and what came back is the state it acts on. When a session's history is not in memory, after a restart or once it was evicted, the server restores it from the SSD session cache, and if the disk does not hold it either, prefills it in full. On one R9700, after a server restart, a resumed 45K-token agent session restored its history from disk and got its first token in 2.6 seconds, against 56.2 seconds of full prefill, with an answer byte-identical to full prefill's.

Limits and what's next

• Closer to free speed. Speculative decoding makes generation faster without changing a single output token: the speed comes for free. PFlash is not there yet, since it still gives up some answers for its speed-up, as on LongBench-v2 and in late chat turns. The plan is to make it more robust where it fails today, so that the speed-up costs as little quality as possible and comes closer to that idea.
• Long chats lose a few late answers. On MTRAG PFlash answers 189 of 255 turns against 197 for full prefill, and the gap sits in the later turns. How much a follow-up recalls is set by θ: a lower value recalls more and may answer better, at some cost in follow-up time. We chose 7.5% on a development replay and have not tuned it on any benchmark here. Next we measure θ on held-out multi-turn chats, starting with 5%, and report the trade-off between late-turn answers and follow-up time instead of a single number.
• Follow-ups are slower than a warm cache. 5.4 seconds against 2.5 for full prefill on the R9700, because each new question is scored on the drafter and what it recalls is prefilled. Over a whole chat PFlash still comes out ahead (85 against 160 seconds), because the long turns it speeds up cost far more.
• Nothing below 16K. PFlash leaves alone any prompt or new message under 16K tokens. We set that threshold on development data before running the benchmarks here: below 16K, shortening lost information that full prefill keeps, and the seconds it saved against full prefill did not make up for it. Bringing PFlash to shorter prompts will take a revised approach that does better on both, not just a lower threshold.
• Up to 220K tokens. With PFlash on, the engine takes prompts of up to 225,280 tokens and refuses longer ones: above that, the drafter's scoring pass runs out of memory on a 32 GB R9700.
• Batched serving. PFlash runs on the engine that serves one request at a time. Lucebox's default launcher serves several sessions at once, so the next step is to implement PFlash in the batched engine.
• Other models. Every number in this post uses Qwen3.8-27B as the answering model. Next is DeepSeek V4, which Lucebox also serves: put this PFlash in front of it and run the same benchmarks, starting with whether the same budget holds for a different model.

Resources

• Weights. The three small models PFlash adds on top of the drafter are in the PFlash collection on Hugging Face: the segment probe, the request probe and the head scorer.

Related

• PFlash: 10× prefill speedup over llama.cpp at 128K on a RTX 3090
• Qwen3.8-27B on the AMD R9700: up to 227 tok/s
• Luce KVFlash: 256K context with 72 MiB of KV on the GPU