Is Your Local Compute Wasting Free Throughput?

actual work: two simple flags not only quadrupled my tok/s but also doubled my kv cache for free; here's the proof so you can do the same

PROLOGUE

If you are already running your own models on your own hardware (local, on prem, your metal, etc), then you are likely leaving some amount of free compute or ‘spend’ on the table without realizing. If you aren’t, why not just start for fun?

your owned compute

I put together this entire test, created the scripts, watched the numbers as it ran, loaded the VRAM, gathered the screenshots, made the graphs, etc.all so you can recreate and see for yourself. Amazing stuff here.

---

FORWARD

Read these gems first

---

LETTER

One vLLM flag took my 5090 from 75 to 350 tok/s (tokens per second or ‘tps’ like my test refers) on code editing, 4.65×, zero extra VRAM, and I diffed every output against baseline: byte identical. The same flag made creative writing 5% *slower*.

A second flag doubled my KV cache capacity for free. Did you read that? FREE. And along the way I accidentally proved that temperature 0 doesn’t mean deterministic.

Receipts, controls, and repro commands below.

made this graph for you guys from my own data so you wouldnt have only the cli to stare at

---

Everything ran on 1 machine, one model, loaded once, SPECS:

1. NVIDIA GeForce RTX 5090, 32 GB VRAM, driver 580.173.02, stock 575 W limit

2. Qwen3 32B AWQ (4 bit [not even 8 bit haa], ~19 GB of weights)

3. vLLM 0.20.0, PyTorch 2.11 (CUDA 13)

4. max-model-len 16384 --max-num-seqs 32 --gpu-memory-utilization 0.90 --seed 0`, prefix caching disabled, temperature 0, thinking disabled

---

The prompt set tested are 24 fixed prompts across 3 deliberately different ways to use intelligence:

1. Code Editing. “Here’s a Python file, make a small change, return the complete file

2. Extractive RAG. “Answer from a provided passage, quoting sentences verbatim

3. Creative Writing. “Short stories and poems (i gave nothing in the prompt to copy)

---

That split isn’t random or too hard to understand… it *is* the experiment,

Spec decoding normally needs a second, smaller model drafting tokens for the big one to verify. vLLM’s n-gram (prompt lookup) speculation skips the second model entirely.

It drafts by searching the *prompt itself* for matching n-grams and proposing whatever followed them last time. No draft model. No extra VRAM.

One flag: --speculative-config '{"method": "ngram", "num_speculative_tokens": 5,
                       "prompt_lookup_max": 4, "prompt_lookup_min": 2}'

The theory was that this should fly when output copies input, and do nothing when it doesnt. Maybe 30% increase would have been insane to me…

It exceeded my expectations…

Look at this

Code hit 364 tok/s, RAG hit 269 tok/s after baselining only 75 tok/s…NUTTY

Median decode speed at batch 1. Time to first token (TTFT as i say) was unchanged everywhere, speculation doesn’t touch prefill, and it’s a nice internal sanity check that it didn’t.

Global draft acceptance: **51.3%** (2,635 of 5,140 drafted tokens), and the speedups track exactly where acceptance should be high. The bottom row is the honest one: on creative writing i’m drafting tokens that never get accepted because its open ended on purpose to test, and that costs about 5%.

---

Now..The part everyone skips… proving “free”. A speedup without a correctness check is marketing. Spec decoding is supposed to be “lossless” meaning rejected drafts get discarded and recomputed, so output should match plain decoding exactly. I tested it instead of trusting it..

Two identical baseline runs → **24/24 outputs byte identical**. The harness is deterministic, so any diff that appears later is signal, not noise, and, every code and RAG output, the 16 prompts where the speedup actually lives, matched **byte for byte**

1 hole: for creative writing **Speculation on vs. off:** **18/24 identical** which was the most interesting thing i found. So their claim survives where it matters but i did find 6 byte divergences myself.

All of those were the creative writing prompts, drifting mid generation. Spec decoding isn’t “lossy”; rejection sampling mathematically guarantees the output distribution. What changes is the batch shape of the forward pass when drafts get accepted.

For Code and RAG here, every next token is strongly determined, so a nanoscale nudge changes nothing, thats why those came back 16/16, byte identical. Creative prompts at temperature 0 are full of near ties between candidate tokens, and a near tie flips on a nanoscale nudge. Greedy decoding pins the sampler, not the math. So really this is truly incredible.

My precise claim then: **byte identical on exactly the workloads where it is fast; distribution equivalent elsewhere.**

Ill take that trade every day. (But “temp 0, same seed, diff output” is worth knowing about before you build tests that assume otherwise.)

---

Bonus, kv cache DOUBLED. Literally just:

‘—kv-cache-dtype fp8’

…HALVES the memory EACH cached token needs. WHAAAAT??@!

Its real…

KV cache went from **33,760 to 65,536 tokens, 1.94×** at the same memory budget, straight from vLLM’s startup logs. Double the concurrent conversations, or double the context headroom, on the same 32 GB.

+1–2% at batch 1, +3% at 16 way concurrency (635.9 → 656.0 tok/s aggregate). Noise, and confounded anyway, since FP8 KV also switches the attention backend from FlashAttention to FlashInfer.

12/24 outputs drifted from baseline (all 8 creative, 2 code, 2 RAG). The near tie mechanism predicts drift, and drift is not damage, but unlike speculation, FP8 KV carries **no losslessness guarantee**, and quantization error compounds with sequence length. Until I run perplexity plus long context retrieval evals, the honest verdict is: **capacity gain proven, quality cost unmeasured.** Treat it as a capacity feature, not a speed feature.

---

so…the Juice? Your GPU mostly runs coding agents, code editing, RAG, summarization, extraction…seeing anything where output echoes input? TURN ON N-GRAM vLLM SPECULATION.

Go look at my baseline numbers again…75 tok/s to 300+ with just TWO SIMPLE LINES OF CODE???

In my measurements, it is the closest thing to free performance in local ai: no VRAM, provable fidelity on these workloads, 2.4-4.65x increase on decode which is just cartoonish considering its “FREE”.

If you are using mostly open ended chats or creative writing generation, skip it and opt for prefix caching.

The FP8 KV x2 across the board doubles anyones headroom now; just run some tests first if you are using any sensitive data.

---

REPRODUCE THIS

The harness is a few small scripts: fixed prompt set, streaming benchmark client, cold server restart between configs, and raw per request JSON out the other end; including every full output text, so the byte identical claim is checkable with `diff` rather than taken on faith.

**Limitations, stated so nobody has to find them for me:** one GPU, one model, one quantization, 24 prompts, batch 1 decode focus, single run per treatment (defensible bc the baseline repeat was 24/24 deterministic), FP8 quality eval still open. The acceptance rate vs speedup *pattern* is what I’d bet generalizes; the exact 4.65× is my machine and my prompts.

---

FIN

What do you think? Have you tried it?

---

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY