In 2026, the entire AI industry has been racing towards one capital goal: build bigger data centers, buy more chips, burn more electricity. Aka horde compute. Google’s cloud grew 1300% in the last year alone. (that will certainly die, but still…)

This Stanford research paper [ Stanford Research Hazy Studio Paper ] written by Chris Re and legendary chip architect John Henessy, makes the argument that cuts against that entire industry strategy:

a huge share of what people actually use AI for doesn’t need any of that.

FORWARD

It can run on ordinary local hardware. Sounds a lot like my previous letters: [Is Your Junk Drawer a Supercomputer? [2025]] go read these, I figured this would happen. [Is Compute the New Oil? [2025]] , Rule #1: [VRAM > Unified Memory [2026].

Subscribe now

Historical precedent. Before personal computers existed, all computing lived on massive shared mainframes, because nothing smaller could do the job. That changed not because PC’s became more powerful than mainframes, but because chip efficiency kept doubling. Roughly every 1.5 years. Until…local machines became good enough to handle real work without the cost of a shared, giant system.

Once that threshold was crossed, computing power redistributed away from centralized mainframes permanently. Entire industries were built on that shift (Microsoft, Apple, Dell) aka the whole PC economy you know today. While the former mainframe giants (IBM’s bread and butter) shrank into a niche.

The paper’s argument: AI compute is approaching that same threshold right now… It is a major claim with data, that argues where economic value and infrastructure spending should be heading.

Most AI demand is simple, which, can be a problem. Data cited in the paper shows 77% of everyday LLM requests are simple things: writing help, general advice, quick lookups. Not frontier level reasoning.

Yet, the industry routes nearly all of this demand through the same expensive infrastructure built for the hardest 5% of problems.

To me, that is an efficiency mismatch at a massive scale. It means a large fraction of the capital being poured into new data centers may be overbuilt for the workload that is showing up in reality.

What the researchers actually tested. To check whether local ai could actually absorb simpler demand, they ran a large controlled test: over 1 million queries, across 20+ small ai models (Qwen3, Gemma, IBM Granite etc.) and 8 types of chips: 3 consumer grade (i wonder if 3090’s?) and 5 enterprise grade (probably 5 RTX 6000s).

The results. Local models correctly handled 88.7% of everyday chat and reasoning questions, meaning most typical ai demand doesn’t structurally require a data center at all. Local model accuracy is up 3.1x from 2023-2025 (absolutely way more in 2026 that isnt measured). A group of local models working together, beat cloud routed ai on 3 of 4 benchmarks. Local chips are still catching up in efficiency about 1.5x behind for now. Local ai efficiency shown to improve 5.3x in those same 2 years.

They also introduced a new measurement system: Intelligence Per Watt (IPW): how much correct outpute you get per watt of power used.

Clearly indicating a deliberate shift in what the industry should be optimizing for…

This is a bigger story than just devices. If a large share of AI workloads can shift to a local/edge hardware, the implications go well past consumer gadgets:

Capex risk. Hundreds of billions in planned data center spending assumes demand keeps centralizing. If a meaningful chunk of demand redistributes to local instead, that changes the ROI math for hyperscalers.

Energy and grid strain. Every workload that runs locally instead of in a data center is one less strain on power grids that are already being stretched by ai buildouts. A policy and infrastructure issue, not a hypothetical.

Competitive landscape shift. This is good news for chipmakers focused on efficient local silicon (Apple, Qualcomm, AMD) relative to the current “sell more giant GPUs to data centers” model that NVIDIA has ridden. It’s the mainframe to PC story with new players in the mainframe and PC roles.

Business model pressure. Companies charging per API call for cloud AI face losing entire business models if good enough local alternatives are free to run once you own the hardware. People will not spend $3k+ a year for intelligence if they can build a pc that does it for them.

The paper’s data stops in 2025. Imagine if this was released now, after the advent of hermes getting nvidia and linux support, plus models like GLM 5.2, the king of 3090’s: qwen 3.6, etc. Every point would hit 10x harder.

These are the things I am interested to see the paper expound on after 2026 is recorded history:

New local chips keep shipping with more on device ai throughput per watt. So the hardware is already adapting to the paper’s argument in a way.

Small open models keep closing the gap with frontier models faster than frontier models can even pull away anymore. Eventually…it won’t even be a measurement, or a thought, just run it local.

Facts. Some reasonable local unimems can already run models approaching 200 billion parameters, but, I say that to say: VRAM is laughing at those numbers if unimem can pull it off, your GPU can scrape more. A smartphone NPU can run a 7B model at 8+ tok/s, which is actually usable real time ai, on a phone chip. The research had direct collaboration and input from SambaNova, Ollama, Snorkel AI, IBM, OpenRouter, AMD, TogetherAI aka actual infrastructure and chip companies, not just a university lab.

The team also shipped a tool. The open sourced, hardware agnostic, benchmarking harness, so anyone can measure their own “intelligence per watt".” They are saying, hey…it isnt just theory…

Cant wait to watch these cloud companies try to cope with this in live time haha..

So, if its already the case that over 80% of ai usage could be done and served by a local model on older hardware, it doesnt make a lot of sense to keep building data centers everywhere because they can’t all operate in the green with this kind of progression.

Don’t send the simple prompt to a billion dollar data center if your old laptop would be happy to do it and keep your data local. If you fall outside of the 88% (like myself and most builders), its still a signal to go local instead of subscriptions.

Share

What do you think?

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

That shows real builders you are actually the exact opposite. Someone has to say it. I remember even just random twitter encounters humbled me and I was thankful for it because I then learned (always learning).

Gonna name some common names: new Steam Machine, Apple Mac Mini, NVIDIA DGX Spark, and AMD’s new spark competitor box that also comes in at like $3k). These are poopy devices to buy for the price and you dont look cool because you got openclaw leaking your data everywhere on it. DON’t BUY lol.

Instead…do something like this:

A lot of you guys got rekt into buying noticeably worse performing hardware for agents by opting out of GPUs and into bad marketing (mac minis, DGX Spark, aka any device utilizing unified memory instead of VRAM). OpenClaw was worth testing an old mac mini or something but not to roll out across your enterprise without any security functions like good loorrrd.

So, let me explain why using a GPU to serve your local weights on its VRAM is always going to be better, for cost and performance, at least for the next 5 years.

Subscribe now

Tired of seeing open mouth youtube accounts gain a tick off confusing real people trying to enter.

Why?

Well…

The median builder workload punishes the exact thing unified memory is bad at. It also rewards the exact thing VRAM is good at. Few reasons ill get into.

Prompt processing is their daily pain, not really token generation. They are feeding large context constantly. Think agent loops that re ingest history every turn. That reading phase is compute bound and unified memory machines have weak GPUs. Horror story always the same here, something like: “m series holds the model fine but takes 90+ seconds before the first token on a long prompt.” On a real GPU with VRAM, like i am saying, it takes seconds. That gap is the difference between usable and unusable.

Bandwidth, tok/s directly. Token generation is bandwidth bound so not the same as compute. GDDR7//HBM (~1.8 TB/s on a RTX 6000, ~936 GB/s even on an old 3090) vs LPDDR5x (~256-819 GB/s) is a 2-7x gap that shows up as your tok/s. Builders post the tok/s….so, they buy the thing that makes the number go up. Simple.

VRAM does more than inference. Serving multiple users and fine tuning both need FLOPS, not just a big memory pool. Unified memory can hold a 70B but crawls through a training backward pass or chokes under concurrent load. Builders want one box that runs AND trains AND serves. That is GPU sillicon, NOT poopy unified memory.

Tinkering cultures rules. The hobby/career is modular. You start with a used 3090 or 2080, add a second, swap, overclock, resell. Unified memory is literally SOLDERED AND FIXED at purchase, at literal comparable prices to USED GPUs that actually RUN WHAT YOU THINK YOU ARE BUYING. Bought the wrong RAM tier on unified memory? You’re stuck. A used 3090 is around $1,000 right now and is still the cheapest on ramp there is to local ai.

CUDA. People scoff here, but they forget… Almost every tool a builder touches in this realm (vLLM, exLlama, llama.cpp’s best kernels, FlashAttention, the newest quant formats [NVFP4, AWQ, GPTQ], speculative decoding, DFlash, MTP) ships on CUDA FIRST, often CUDA only, and is debugged there first. On a Mac or AMD, you are perpetually a few months behind, hitting “not supported on this backend” walls, or running the slower path. Builders optimize for the “new trick works the day it drops” for content and to stay at the tip of the spear, so that is NVIDIA specific GPU’s, not an argument for the unified mem DGX spark. The net result is that unified memory makes you a second class citizen in the software that matters.

NVIDIA GPUs hold value. You can recoup a lot of your spend on the next upgrade. A soldered box can’t be parted out; when you are done, you sell one thing for one hugely depreciated price.

The GPU is multi purpose. With a GPU, you can do gaming, rendering, video, all things a unified memory box literally cannot do, so a GPU earns its keep even when its not running your intelligence. To me, that is just case in point.

Performance per dollar is about work completed, not sticker price. Even when two machines cost the same, the GPU (remember, VRAM lives on the GPU) finishes the job FASTER. Faster prompt ingest, more tok/s, real concurrency. So, the effective cost per task is lower. You are not paying for hardware, you are paying for THROUGHPUT; VRAM gives you more throughput per dollar spent, and its not a debate.

You can’t turn unified memory into an API endpoint. Remember the bottleneck: to generate one token, the GPU has to stream all the model’s weights through the chip once. That read is the slow, expensive step. So, if you ever decide to serve or batch from your VRAM you can. The GPU can give you more concurrent users because every active request needs its own KV cache to be actually HELD in VRAM. More GPUs = More people that can chat and have access with your local intelligence (an impossibility with unified mem alone). Also, vLLM opened the door to continuous batching, so the requests arrive and finish at different times. vLLM slots new ones into the batch the instant a seat frees up, keeping the bus full and always optimized. This needs both spare compute (unified memory fails) AND spare VRAM to juggle caches!

Serving one user wastes 95% of the GPU. Serving thirty fills the idle compute for almost free. Same weight read, same watts, thirty times the output. That’s the whole reason a GPU pays for itself and a unified memory box can’t: filling the bus only works if the bus has empty seats, and only real GPUs have them! This is why GPU farms existed in the first place.

The used market is a cheat code for VRAM, not unified mem. A flood of ex mining and old gaming 3090s sell for like $900-$1100 (if nice or refurbished shape) for 24 GB of 936 GB/s memory. Nothing in unified memory land comes within range of that bandwidth per dollar. There is no “used mac studio m3 ultra at a third of retail” market haha; soldered machines depreciate as one sealed unit. Go get a used NVIDIA 3090 if you are trying to jump in the arena.

Stop getting rekt.

What do you think?

Share

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

So, before an agent can even answer a prompt, something has to find the handful of relevant pages and hand only those over. Essentially, every complex dataset should have its own personal librarian ready to supply the agent/LLM with the relevant files to execute the prompts you give it, in the background.

This whole technique has already been coined & known as RAG (Retrieval Augmented Generation) but you can think of it as: find the right pages first, then let the agent write the answer.

This is why RAG is big and certain models solidify integrating the right transformer into your production and its set for a long time. Sentence-trasnformers all-MiniLM-L6-v2: this is the most downloaded model on all of huggingface for a reason, and this letter unpacks why and how its used.

Think of it as hiring a very fast librarian. Your mind should expand to think about all the things you never knew were possible without RAG especially in the agentic world.

The first stage is just to set up the library. Done once. You cut all your documents into small chunks, about a paragraph each. The model reads each chunk and gives it a kind of “fingerprint”. A code that captures what the paragraph is about, not just the words in it.

All those fingerprints get filed away in a special cabinet built for this (called a vector database).

Now, every paragraph you own has been quietly tagged by meaning.

Second stage, someone asks a question in live time: the model reads the question and give it a fingerprint too. It compares that fingerprint against everything in the cabinet (vector DB) and pulls out the few paragraphs that match closely, or, the prints that most likely hold the right answer or data.

Those few paragraphs get handed to the agent or LLM along with the question, essentially saying “Here’s all the relevant material, now answer the user’s prompt.” Think of a librarian helping someone with the Dewey Decimal System search, except its an agent doing that for your mountains of data you wouldnt want to read through yourself.

The agent or LLM will then respond based on real, specific paragraphs instead of guessing. This whole technique has obviously been coined RAG (Retrieval Augmented Generation) but you can think of it as: find the right pages first, then let the agent write the answer.

Every app that says you can chat with your data, is really two halves: find the right pages (RAG half) then write the eloquent answer (agent/LLM half).

That little model above is my favorite for this because of its functionality and smaller size. It functions as the finder, not the talker. It is the librarian who quietly pulls the three right pages off the shelf so the famous agent can do the talking.

It will never get the credit, but almost nothing works without it.

So why is that model above the literal most downloaded AI model out of everything in existence right now?

Its small and cheap. Runs fast on an ordinary computer. This is why builders reach for it without hesitation.

Its good enough. For most real world jobs, its results are plenty accurate.

It is the default. This is the bigger one. The most popular free tools and tutorials for building these apps use this exact model as their built in starting point. So every time a dev follows a tutorial, runs a test, or rebuilds their app, this model gets pulled down automatically (usually over and over).

Last point, think of LangChain. If you are a builder you may have heard of it, if not, you probably haven’t. Either way, good example.

You hand LangChain a stack of documents, say, your company’s 500 page handbook.

LangChain will be the thing chopping it into pieces or fingerprints, these are the chunks I mentioned. For each one of those chunks, LangChain calls all-MiniLM-L6-v2 and says “give me the meaning fingerprint of this paragraph.” The model hands back that vector in the form of a list of numbers.

LangChain then takes all those fingerprints and files them in a vector database we mentioned earlier. Lets say an employee asked “How many sick days do I get?”

Once the user starts prompting, LangChain runs the question through the exact same model to get its own fingerprint for the prompt.

The database hands back the 3-4 most relevant chunks (the ones about the user’s sick leave question) even if the employee handbook never uses the words “sick days” and instead says “paid medical absence” (That is the meaning/fingerprint matching at work)!

Langchain then takes those retrieved paragraphs and staples them onto the question before sending it to the Agent/LLM, roughly like: “using the following text from the handbook: […the relevant paragraphs…], answer the question: how many sick days do i get”? This is all happening in the background, which is why its basically used by everyone and thanked by no one building in the AI realm.

The all-MiniLM-L6-v2 is the translator at both ends: it turns your documents into fingerprints once, and turns each incoming question into a fingerprint so it can be matched against them with something like LangChain before it hits your agent.

LangChain is the conveyor belt moving everything between the translator, filing cabinet and the agent.

Adding LangChain + all-MiniLM-L6-v2 = your very own data librarian.

Subscribe now

What do you think?

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

With all the propaganda being spewed about AI, data centers, energy prices, etc., I couldn’t help but wonder: why the world’s data is not just underwater by now. If it were the case that we could achieve better results underwater, without building more on land, then that should be ‘Plan A’.

Once I saw 30,000 acre plots being used to essentially spin GPU’s (Wyoming), I wanted to study what the org’s actual documentation argued for, both from Project Natick (Microsoft) & Project Hamina (Google), to then see what their own signals were to build more data centers onshore.

Subscribe now

This letter is the fruit of that study…

First principles here are physics, depreciation schedules, TCO structure and what these orgs actually disclose in filings.

Here are the REAL reasons:

The Submarine Hull Couples a 25 Year Asset into a 3 Year Asset

This is a pure accounting killer, not engineering.

In a data center on land, the shell normally lasts about 25 years while the hardware inside refreshes on a 3-6 year life cycle. (Amazon recently just shortened their own server useful life from 6 years to 5 years because AI churns hardware faster.)

A sealed pressure vessel welds those two depreciation curves together: whatever you seal in is what you have until a ship retrieves it.

Sealing ~$300-$500M of accelerators (kind of a modest AI deployment) into hulls means accepting that the asset is at ~30% of book value when you can next touch it, with zero mid life densification. Kind of obvious why CFO’s wouldn’t want to sign that.

The land megacampus is, financially, a vehicle for decoupling long term capital vs. short term capital; that is why it wins.

Cooling was Never the Cost Driver

An actual TCO stack of an AI datacenter will look something like this: Hardware costing ~60%-70% of total cost of ownership; the GPUs in one NVL72 rack cost costing ~$3M, several times the fully built, shell and power cost of the floor space holding them.

Cooling overhead at hyperscale is the gap between Hamina’s PUE ~1.10 and Natick’s ~1.07. So, the ocean’s entire value proposition attacks only ~3% of ~30% of the cost stack, under 1% of TCO, while degrading the 70% (i.e., the hardware).

Hyperscalers solved their own problem on land before Natick could matter. The experiment’s premise was arbitraged away by its own sponsors during the experiment.

Rack Power density Outran the Hull’s Physics

Natick Phase 2 was 12 racks at ~240 kW, passively conducting head through the hull to seawater.

Current AI racks are 130 kW and NVIDIA’s announced Rubin Ultra generation targets ~600 kW/rack.

You cannot move 600 kW/rack through a hull wall by convection; you need direct to chip liquid loops and large engineered heat exchangers inside the vessel (which, turns out to be the same cooling plant as on land). The ocean as a passive heatsink only works at power densities the industry abandoned.

This is why Microsoft’s takeaway was “immersion cooling onshore”: once you are doing the liquid loop engineering anyway, the seawater does not add anything. But, for Project Hamina, where Google uses the Baltic Sea to cool data centers on land, this approach is more promising under this specific point of hull physics.

Training Models is the Most Centralizing Workload in Computing History

A training cluster needs every GPU inside a single low latency, multi terabit, grid fabric; this is why the industry builds single 100 MW+ halls with hundreds of thousands of short run optical links.

Underwater pods are the opposite, replacing one 300 MW hall requires ~1,000 Natick class vessels, each its own network island connected by individual subsea cables. The fabric you’d need between them does not exist at any price, and the speed of light through that much fiber breaks synchronous training regardless.

Natick was designed for the 2015 workload: stateless, latency tolerant cloud services, etc. Hamina is still running with better results from adding wind at a point in the bottleneck.

The world pivoted to the one workload that structurally cannot be sharded across pods (AI/Agents). Which is why those same wins in 2015, that spurred the underwater experiment to begin with, is not longer even the meta. The seabed didn’t lose to land; it lost to the reduction focused operation.

There is No Marine Industrial Base for $400B/yr

Hyperscaler data center apex is running at hundreds of billions a year. That money is more absorbable on land because the supply chain already exists.

The global marine heavy construction industry: vessel days, dynamic positioning ships, subsea welding capacity, is sized for oil, gas, offshore wind and could potentially absorb single digit billions of data center work per year. My prediction is that more companies will want to go underwater now that we can run way larger param models on much smaller hardware.

Even at max enthusiasm, underwater could not have been where the money went; the deployment pipe is two orders of magnitude too narrow. Capital flows to whatever can absorb it at the required velocity, and the #1 binding constraint since 2023 is the time needed to compute, not efficiency.

Every term in the equation moved against the ocean between 2018 and 2023.

The one bet that could still flip an era is wind colocated pods (attacking power, the real constraint, rather than cooling) and that is precisely the variant being attempted as we speak…

Share

What do you think?

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

Share

Every breakthrough in AI looks like one moment, but it usually comes from two separate groups of people who never talk to each other.

The first group writes papers and ships open source code. They are usually employed by a research lab, they publish on arXiv, and the work is freely a…

Read more

Read more

So much still to unpack just from the agent conference, let alone trying to keep up with the lightspeed pace of AI running on more affordable hardware. (Unrealized local potential from models like Gemma 4b, Qwen 27b..)

But one of my most memorable encounters from the conference, (read this letter if you are not up to…

Read more

Subscribe now

Read more

The setup is simple to describe and hard to believe until you see it working.

Share

You deploy one AI agent into your company Slack. You…

Read more

Last year, 300 people showed up. This year, over 2,000 are expected. Every sponsorship package is sold out. The speaker list reads like someone merged the org charts of Thomson Reuters, GitHub, OpenAI, Bloomberg,…

Read more

It wi…

Read more

This isn’t a hot take. It’s something you can watch happen in real time, twice, in the span of a single week.

Read more

Read more

Before we start, I wanted to deeply thank the newest paid subscriber to The Gug Letter, @ Chad Cummings! Greatly appreciate you!

Chad and I went to law school together. He is by far the smarter one between us & also writes an incredible Law Journal worth reading.

Really enjoyed this recent edition: Tax Consequences of Using a Single-Member LLC as …

Read more

Let me explain.

Read more

I didn’t cut any corners, which means you won…

Read more

DATA

Removing all trash signals, here is what the data says: …

Read more

---

Last August, I wrote that generative AI would bring software to $0: Will Generative AI Bring Software to $0?. The…

Read more

The silicon that powers ChatGPT is the same silicon that runs Call of Duty. That's not…

Read more

Today is the Nativity of Our Lord, Christ is Born! Happy New Year and Happy Holidays to everyone!

I am excited and blessed to build more in 2026.

Back to our advertised schedule of 2 free letters and 1 paid letter each week. Want to make sure each letter is value packed and not just ai slop for you guys.

Share

I thought it would be fittin…

Read more