If you are already running your own models on your own hardware (local, on prem, your metal, etc), then you are likely leaving some amount of free compute or ‘spend’ on the table without realizing. If you aren’t, why not just start for fun?

Subscribe now

I put together this entire test, created the scripts, watched the numbers as it ran, loaded the VRAM, gathered the screenshots, made the graphs, etc.all so you can recreate and see for yourself. Amazing stuff here.

FORWARD

Read these gems first

LETTER

One vLLM flag took my 5090 from 75 to 350 tok/s (tokens per second or ‘tps’ like my test refers) on code editing, 4.65×, zero extra VRAM, and I diffed every output against baseline: byte identical. The same flag made creative writing 5% *slower*.

A second flag doubled my KV cache capacity for free. Did you read that? FREE. And along the way I accidentally proved that temperature 0 doesn’t mean deterministic.

Receipts, controls, and repro commands below.

Everything ran on 1 machine, one model, loaded once, SPECS:

1. NVIDIA GeForce RTX 5090, 32 GB VRAM, driver 580.173.02, stock 575 W limit

2. Qwen3 32B AWQ (4 bit [not even 8 bit haa], ~19 GB of weights)

3. vLLM 0.20.0, PyTorch 2.11 (CUDA 13)

4. max-model-len 16384 --max-num-seqs 32 --gpu-memory-utilization 0.90 --seed 0`, prefix caching disabled, temperature 0, thinking disabled

The prompt set tested are 24 fixed prompts across 3 deliberately different ways to use intelligence:

1. Code Editing. “Here’s a Python file, make a small change, return the complete file

2. Extractive RAG. “Answer from a provided passage, quoting sentences verbatim

3. Creative Writing. “Short stories and poems (i gave nothing in the prompt to copy)

That split isn’t random or too hard to understand… it *is* the experiment,

Spec decoding normally needs a second, smaller model drafting tokens for the big one to verify. vLLM’s n-gram (prompt lookup) speculation skips the second model entirely.

It drafts by searching the *prompt itself* for matching n-grams and proposing whatever followed them last time. No draft model. No extra VRAM.

One flag: --speculative-config '{"method": "ngram", "num_speculative_tokens": 5,
                       "prompt_lookup_max": 4, "prompt_lookup_min": 2}'

The theory was that this should fly when output copies input, and do nothing when it doesnt. Maybe 30% increase would have been insane to me…

It exceeded my expectations…

Look at this

Median decode speed at batch 1. Time to first token (TTFT as i say) was unchanged everywhere, speculation doesn’t touch prefill, and it’s a nice internal sanity check that it didn’t.

Global draft acceptance: **51.3%** (2,635 of 5,140 drafted tokens), and the speedups track exactly where acceptance should be high. The bottom row is the honest one: on creative writing i’m drafting tokens that never get accepted because its open ended on purpose to test, and that costs about 5%.

Now..The part everyone skips… proving “free”. A speedup without a correctness check is marketing. Spec decoding is supposed to be “lossless” meaning rejected drafts get discarded and recomputed, so output should match plain decoding exactly. I tested it instead of trusting it..

Two identical baseline runs → **24/24 outputs byte identical**. The harness is deterministic, so any diff that appears later is signal, not noise, and, every code and RAG output, the 16 prompts where the speedup actually lives, matched **byte for byte**

1 hole: for creative writing **Speculation on vs. off:** **18/24 identical** which was the most interesting thing i found. So their claim survives where it matters but i did find 6 byte divergences myself.

All of those were the creative writing prompts, drifting mid generation. Spec decoding isn’t “lossy”; rejection sampling mathematically guarantees the output distribution. What changes is the batch shape of the forward pass when drafts get accepted.

For Code and RAG here, every next token is strongly determined, so a nanoscale nudge changes nothing, thats why those came back 16/16, byte identical. Creative prompts at temperature 0 are full of near ties between candidate tokens, and a near tie flips on a nanoscale nudge. Greedy decoding pins the sampler, not the math. So really this is truly incredible.

My precise claim then: **byte identical on exactly the workloads where it is fast; distribution equivalent elsewhere.**

Ill take that trade every day. (But “temp 0, same seed, diff output” is worth knowing about before you build tests that assume otherwise.)

Bonus, kv cache DOUBLED. Literally just:

‘—kv-cache-dtype fp8’

…HALVES the memory EACH cached token needs. WHAAAAT??@!

Its real…

KV cache went from **33,760 to 65,536 tokens, 1.94×** at the same memory budget, straight from vLLM’s startup logs. Double the concurrent conversations, or double the context headroom, on the same 32 GB.

+1–2% at batch 1, +3% at 16 way concurrency (635.9 → 656.0 tok/s aggregate). Noise, and confounded anyway, since FP8 KV also switches the attention backend from FlashAttention to FlashInfer.

12/24 outputs drifted from baseline (all 8 creative, 2 code, 2 RAG). The near tie mechanism predicts drift, and drift is not damage, but unlike speculation, FP8 KV carries **no losslessness guarantee**, and quantization error compounds with sequence length. Until I run perplexity plus long context retrieval evals, the honest verdict is: **capacity gain proven, quality cost unmeasured.** Treat it as a capacity feature, not a speed feature.

so…the Juice? Your GPU mostly runs coding agents, code editing, RAG, summarization, extraction…seeing anything where output echoes input? TURN ON N-GRAM vLLM SPECULATION.

Go look at my baseline numbers again…75 tok/s to 300+ with just TWO SIMPLE LINES OF CODE???

In my measurements, it is the closest thing to free performance in local ai: no VRAM, provable fidelity on these workloads, 2.4-4.65x increase on decode which is just cartoonish considering its “FREE”.

If you are using mostly open ended chats or creative writing generation, skip it and opt for prefix caching.

The FP8 KV x2 across the board doubles anyones headroom now; just run some tests first if you are using any sensitive data.

REPRODUCE THIS

The harness is a few small scripts: fixed prompt set, streaming benchmark client, cold server restart between configs, and raw per request JSON out the other end; including every full output text, so the byte identical claim is checkable with `diff` rather than taken on faith.

**Limitations, stated so nobody has to find them for me:** one GPU, one model, one quantization, 24 prompts, batch 1 decode focus, single run per treatment (defensible bc the baseline repeat was 24/24 deterministic), FP8 quality eval still open. The acceptance rate vs speedup *pattern* is what I’d bet generalizes; the exact 4.65× is my machine and my prompts.

FIN

What do you think? Have you tried it?

Subscribe now

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

This is a giant month for Bitcoin & Crypto.

Washington is currently sitting on the most important crypto law ever written (CLARITY Act) and the clock runs out in just 3 weeks.

FORWARD

The CLARITY Act, i wrote about this a few years ago when it was becoming a real thing, read these before today:

LETTER

What the CLARITY Act does is simple, even though the politics are not.

It decides, once and for all (supposedly), which government agency regulates which digital asset.

(I mean what does that even mean when the agencies themselves can’t magically understand blockchain enough to regulate it just because a simple law was passed, do they understand it? Can they define what regulation is for an autonomous blockchain ledger?

Probably only in the way that scammers pump meme coins and then call that “crypto”, id say that would be me giving a generous classification of their understanding.)

Bitcoin is not crypto, im becoming more of a maxi every day.

For ten years now, companies in this industry haven’t known whether they answer to the SEC or the CFTC, and that uncertainty (not tech, not demand) has been the biggest thing holding American crypto businesses back. The bill passed its Senate committee back in May with votes from both parties. It has been stuck ever since.

Why? Surely a mix of reasons but a big one seems to be: The President’s own money. I thought this may have been just like a silly hit piece or something but the numbers are there…

The government ethics office released President Trump’s financial disclosure on July 1, 2026. It showed roughly $1.4 billion in crypto related income in 2025, which is more than half of everything he earned.

Democrats are saying that they will not pass a law regulating an industry the President personally profits from unless the law also contains rules preventing officials from cashing in on the assets they oversee.

(Which is kind of funny, because we know Dems do it on a crazy degree just with stocks lmao. Who knows if that is even a good faith argument but thats kind of the sentiment rn.)

Republicans call that a poison pill which isn’t totally wrong either.

My take is that we should not give any agency control over a digital asset without barring those employees from cashing in, IN ANY WAY, on the assets they oversee. BUT,,,,, WE SHOULDNT EVEN BE GIVING THESE AGENCIES CONTROL OVER THE ASSETS AT ALL. To me, its just like, start a lightning network and use BTC, you dont need to be told what your business can accept? Sure it would be easier but why is it needed?

But, this is why neither side has moved.

This past week, the White House planned a meeting between the President and Senators to try to break the deadlock on exactly this section: which, is telling…

Betting markets now put the odds of the passing at about 31% and the Senate leaves for its August recess in roughly 3 weeks. After that, midterm campaigning takes over and the window closes, probably until 2027.

There are two quieter fights inside the same bill worth knowing about.

One is over DeFi: whether the people who merely write open source financial software must register with the government like brokerages do. To me, going after the literal devs who simply build something on top of a blockchain is ridiculous absolutely absurd and no one should actually follow that directive. You cannot wage a war on everything people create, its open source. If they get that wrong, devs will likely pack up.

The other is over stablecoin yield. Whether companies like Coinbase can keep paying interest like rewards on dollar backed tokens. That specific usury angle is Coinbases’s cash cow: worth over a billion dollars a year to them. (COINBASE IS TRASH AND IS QUITE LITERALLY THE OPPOSITE OF BITCOIN’s PROOF OF WORK CONSENSUS)

MEANWHILE, the last crypto law is quietly falling in the follow through.

Last year, Congress passed the GENIUS Act, the stablecoin law, and gave regulators exactly one year: Until July 18th, 2026 (two days ago) to write the FINAL rules…

They missed the deadline...haha

And, how on earth are you really going to start a war with open source by declaring a gov agency the head of regulation over a digital asset? Thats like saying all internet must be controlled by the gov, which, is not going to happen in any reality. (Imo its because they know they can’t just say, all ethereum is under “blank” control because blockchains do not care about borders, its just blatant overreach, how are you going to clearly define what coin is governed by what agency when the agencies themselves are probably still using 1960’s IBM COBOL?) It really is funny.

The proposals exist, the comment periods are still open, but the rules are not done.

Here is the lesson I’d underline twice: passing a law turned out to be the easy part. The agencies charged with implementing these laws are understaffed and behind, and if CLARITY passes, the same thing will happen to it unless Congress also funds the regulators it keeps assigning homework to.

Again highlighting, for the world to see, that Congress and gov do not understand the tech they so desperately want to “regulate”.

While Washington stalls, the market is telling its own story. Inflation came in much cooler than expected this week, the biggest monthly drop in index since 2020. Bitcoin ran from about $62,800 to $65,000 (insanely suppressed paper price btw so take with grain of salt).

The Bitcoin ETFs took in roughly $1.2 billion this week alone. I DO NOT BUY ETF TRASH.

Another super weird wrinkle to watch: gov wallets moved about $288 million of seized Bitcoin and Ether onto Coinbase this week…that kind of move is shady and normally precedes a sale. WHY is the gov moving digital assets to a private company? All while the CLARITY Act is loading??

This is an odd look when we just signaled to the world the importance of owning a strategic Bitcoin Reserve..

The next 3 weeks decide whether the United States gets crypto rules in 2026 or punts to 2027, but the fight is no longer about crypto at all…

Its about whether the law will include ethics RULES that apply to the people passing it.

What do you think?

Share

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

In 2026, the entire AI industry has been racing towards one capital goal: build bigger data centers, buy more chips, burn more electricity. Aka horde compute. Google’s cloud grew 1300% in the last year alone. (that will certainly die, but still…)

This Stanford research paper [ Stanford Rese…

Read more

That shows real builders you are actually the exact opposite. Someone has to say it. I remember ev…

Read more

So, before an agent can even answer a prompt, something has to find the handful of relevant pages and hand only those over. Essentially, every complex dataset should…

Read more

With all the propaganda being spewed about AI, data centers, energy prices, etc., I couldn’t help but wonder: why the world’s data is not just underwater by now. If it were the case that we could achieve better results underwater, without building more on land, then that should be ‘Plan A’.

Once I…

Read more

Share

Every breakthrough in AI looks like one moment, but it usually comes from two separate groups of people who never talk to each other.

The first group writes papers and ships open source code. They are usually employed by a research lab, they publish on arXiv, and the work is freely a…

Read more

Read more

So much still to unpack just from the agent conference, let alone trying to keep up with the lightspeed pace of AI running on more affordable hardware. (Unrealized local potential from models like Gemma 4b, Qwen 27b..)

But one of my most memorable encounters from the conference, (read this letter if you are not up to…

Read more

Subscribe now

Read more

The setup is simple to describe and hard to believe until you see it working.

Share

You deploy one AI agent into your company Slack. You…

Read more

Last year, 300 people showed up. This year, over 2,000 are expected. Every sponsorship package is sold out. The speaker list reads like someone merged the org charts of Thomson Reuters, GitHub, OpenAI, Bloomberg,…

Read more

It wi…

Read more

This isn’t a hot take. It’s something you can watch happen in real time, twice, in the span of a single week.

Read more

Read more

Before we start, I wanted to deeply thank the newest paid subscriber to The Gug Letter, @ Chad Cummings! Greatly appreciate you!

Chad and I went to law school together. He is by far the smarter one between us & also writes an incredible Law Journal worth reading.

Really enjoyed this recent edition: Tax Consequences of Using a Single-Member LLC as …

Read more

Let me explain.

Read more

I didn’t cut any corners, which means you won…

Read more

DATA

Removing all trash signals, here is what the data says: …

Read more

---

Last August, I wrote that generative AI would bring software to $0: Will Generative AI Bring Software to $0?. The…

Read more