Is Your Local Compute Wasting Free Throughput?

actual work: two simple flags not only quadrupled my tok/s but also doubled my kv cache for free; here's the proof so you can do the same

PROLOGUE

If you are already running your own models on your own hardware (local, on prem, your metal, etc), then you are likely leaving some amount of free compute or ‘spend’ on the table without realizing. If you aren’t, why not just start for fun?

your owned compute

I put together this entire test, created the scripts, watched the numbers as it ran, load…