FORWARD

For two years, the bottleneck on local large language models has not been quality. It has been throughput.

You can run Qwen 3.6 or Gemma 4 on a single RTX 3090 (although I would say a 3090 is the absolute floor right now for usable Local AI) and get answers that match frontier class output on most everyday tasks. The problem has been waiting forty seconds for the model to finish writing them (aka latency).

That bottleneck quietly broke in the last three weeks. The technique behind it is called Multi-Token Prediction, or MTP, and vLLM is now the production answer for running it at speed…

Share

This is a working guide to what MTP is, why draft acceptance rate is the only number you should care about, what real users are getting on 24GB, 48GB, 80GB, and AMD ROCm hardware, and how to actually serve it from your own machine today using vLLM.

WHAT MULTI-TOKEN PREDICTION ACTUALLY IS

Multi-Token Prediction is not new in the academic sense. DeepSeek introduced it as a training objective in V3, where the model learns to predict not just the next token but the next several tokens at once through additional lightweight prediction heads.

What changed in 2026 is that those heads are now exposed at inference time and wired into vLLM as a first class speculative decoder under the spec_decode_config flag.

If you have heard of speculative decoding before, the mental model is similar. A small, fast model drafts a handful of tokens, and the big model verifies them in parallel. When the draft is correct, you get those tokens almost for free.

The crucial difference with MTP is that the “draft model” is not a separate model. It is a set of extra prediction heads that were trained jointly with the main model. They live in the same safetensors (ollama can’t run safetensors to my knowledge) weights, they share representations with the main forward pass, and they understand the main model’s distribution natively.

That is why MTP draft acceptance rates in vLLM are landing in the 70 to 85 percent range, where traditional speculative decoding with an ‘off the shelf’ draft model usually struggles to clear 50 percent.

IS MTP LOSSLESS? (THE QUESTION EVERYONE ASKS FIRST)

Yes. This is the most important property of MTP and the single biggest reason it is taking over.

MTP does not change the output of your model. Every draft token is verified by the full main model before it is committed to the response. If the verification disagrees with the draft, the bad token is rejected and the main model emits the correct one instead.

In other words, MTP is a pure speedup. The text you get out of Qwen 3.6 27B with MTP enabled is bit-for-bit identical to the text you would get without it, assuming the same sampling seed and the same sampling parameters.

This is fundamentally different from quantization tradeoffs, where you sacrifice a little quality to fit a model on smaller hardware (think MoE’s). With MTP you sacrifice nothing. You either accept the draft and save a forward pass, or you reject it and pay the original cost. There is no middle ground where quality degrades.

MTP VS TRADITIONAL SPECULATIVE DECODING VS MEDUSA AND EAGLE

Several search queries land here, so it is worth bodying.

Traditional speculative decoding pairs a big model with a separate, much smaller draft model. For example, you might use Qwen 0.5B to draft tokens for Qwen 32B.

vLLM has supported this for over a year. It works, but draft acceptance is mediocre because the small model does not share the big model’s internal representations, and you pay extra VRAM for the second model.

Medusa attaches multiple prediction heads to a frozen base model after training. It is essentially MTP but bolted on after the fact. vLLM supports Medusa. Acceptance is decent but typically lower than native MTP because the heads never participated in joint training.

EAGLE and EAGLE-2 propagate the base model’s hidden states into the draft head. They perform well, and vLLM ships first class EAGLE support, but they require a separately trained draft head that not every model author publishes.

Native MTP, as shipped in DeepSeek V3, DeepSeek V4, Qwen 3.6, and Gemma 4, is the cleanest version of the idea. The prediction heads are trained from the start as part of the same model. They get the highest acceptance rates, they require no second model, and they cost essentially zero extra memory because they were already in the checkpoint.

vLLM now supports all four paths under one unified speculative decoding interface. Native MTP is the one to reach for when the model author trained it, because the acceptance rate ceiling is highest.

WHY DRAFT ACCEPTANCE RATE IS THE ONLY NUMBER THAT MATTERS

When you read MTP benchmarks, ignore the raw tok/s number for a moment. The number to anchor on is draft acceptance rate.

Draft acceptance rate is the percentage of speculated tokens that the main model agrees with on verification. At 0 percent acceptance, you are actually running slower than baseline because of verification overhead. At 100 percent acceptance, you would get a linear speedup equal to your draft depth.

In practice, every accepted token saves you one full forward pass of the main model. So acceptance rate is what converts MTP from a theoretical trick into real wall clock speed.

vLLM exposes acceptance rate in its built in metrics endpoint, which is one of the practical advantages over llama.cpp.

Benchmarks in the last week are landing consistently above 80 percent acceptance on coding workloads with Qwen 3.6 served through vLLM, which is why the 2x to 3x speedup numbers are actually reproducible.

There is also a depth choice. MTP can speculate one token ahead, three tokens ahead, or further. The community has converged on three as the optimal draft depth for current Qwen and Gemma checkpoints. vLLM exposes this directly as num_speculative_tokens. Going deeper increases verification cost faster than acceptance falls off, so you get diminishing returns past depth 3.

Acceptance rate also varies by workload. Code generation, with its highly predictable syntactic patterns, gets the highest acceptance. Creative writing and dialogue typically run 10 to 15 percentage points lower because token-level predictability drops.

WHICH MODELS ACTUALLY SUPPORT MTP IN vLLM TODAY

As of mid May 2026, the production ready MTP enabled models in vLLM are:

DeepSeek V3 and V4. The original implementations. Native MTP support was first class in vLLM before any other engine. Very high acceptance rates, and well suited to multi-GPU tensor parallel deployments.

Qwen 3.6 27B (dense) this is my personal favorite pick for a single 3090. The current sweet spot for 24GB to 48GB consumer GPUs. Officially supported in vLLM 0.9 and later with the native MTP path enabled.

Qwen 3.6 35B A3B (mixture-of-experts). Total parameters are 35 billion, but only 3 billion are active per token. vLLM’s expert parallelism handles MoE routing efficiently, and the MTP heads compose cleanly on top.

Gemma 4 26B. Google’s MTP-trained checkpoint. Fully supported in vLLM with AWQ and FP8 quantization paths.

NVIDIA Star Elastic. Newer, less-tested, but interesting because a single checkpoint can be sliced into 30B, 23B, or 12B at inference time. vLLM has experimental support behind a feature flag.

If you see a community fine tune labeled “MTP Preserved” or “Native MTP,” it means the fine tuner did the extra work to keep the prediction heads intact in the safetensors. Many popular fine-tuning recipes silently strip them.

A QUICK NOTE ON MIXTURE-OF-EXPERTS AND THE A3B NOTATION

The “A3B” suffix on Qwen 3.6 35B A3B trips a lot of people up.

It means “active 3 billion parameters.” The model has 35 billion total parameters, but for any given token, only about 3 billion are actively used. A router network picks which experts to engage per token.

The practical consequence is that VRAM usage is dictated by the total parameter count, while inference speed is dictated by the active parameter count. That is how Qwen 3.6 35B A3B can run as fast as a dense 3B model while having the intelligence of something much larger.

MTP and mixture-of-experts compound nicely in vLLM. The MoE router cuts the per-token cost, MTP cuts the per-step count, expert parallelism distributes the routing across GPUs, and you end up with throughput that would have seemed impossible on consumer hardware a year ago.

THE HARDWARE TIER TABLE

vLLM is GPU-first and designed for serving rather than single stream chat. The tier numbers below assume you are running the official OpenAI compatible server with paged attention and that you want sustained throughput, not just a single quick reply.

24GB VRAM TIER (RTX 3090, 4090, 5090)

Model: Qwen 3.6 27B AWQ 4-bit.

Backend: vLLM 0.9+ with native MTP and num_speculative_tokens=3.

Result: 110 to 135 tokens per second at single-user throughput, 2x to 3x baseline, 64K context window with FP8 KV cache enabled.

Why it matters: this is the first time a single consumer GPU has hit triple-digit tokens-per-second on a dense 27B-parameter model at usable quality, while also serving concurrent requests with continuous batching.

48GB VRAM TIER (DUAL 3090, A6000, RTX 6000 ADA)

Model: Qwen 3.6 27B at FP8 or GPTQ INT4.

Backend: vLLM with tensor parallelism (tp=2 on dual cards) and native MTP.

Result: 2.5x faster inference, 128K to 200K context window, sustained throughput across 8 to 16 concurrent users.

The practical caveat at this tier: when you go above 128K context, switch from FP8 KV cache to unquantized to avoid quality drift in long agent loops.

80GB VRAM TIER (A100 80GB, H100, H200)

Model: DeepSeek V4 or Qwen 3.6 35B A3B at FP8.

Backend: vLLM with tp=1 or tp=2, native MTP, expert parallelism for MoE models.

Result: 2.5x to 3x speedup on DeepSeek, with batched throughput in the thousands of tokens per second range.

This is vLLM’s natural home. The engine was built for exactly this hardware tier, and MTP turns it into the cheapest way to host frontier class open-weight models on your own infrastructure.

AMD ROCm TIER (7900 XTX, MI300X)

Model: Qwen 3.6 27B AWQ or FP8.

Backend: vLLM-ROCm via Lemonade, with native MTP recently added as an experimental path.

Result: Comparable token throughput to the NVIDIA equivalents, with the caveat that MTP support on ROCm is one release cycle behind CUDA.

Why this matters: vLLM-ROCm in Lemonade gives AMD users a real serving path for the first time. If you bought a 7900 XTX or an MI300X and have been stuck on llama.cpp, this is your upgrade.

A note on what is missing from this table.

vLLM does not run on Apple Silicon (dorks, buy a ThinkPad and run Linux). If you are on a Mac Studio or MacBook Pro, MTP is available through llama.cpp on Metal or through MLX nightly builds, but vLLM is not your engine. The rest of this guide does not apply to your hardware. But, you can apply most to llama.cpp or ollama.

vLLM also pre-allocates GPU memory aggressively. It is not a great fit for 12GB cards, where small KV cache budgets defeat the purpose of continuous batching.

If you are on a 3060 or 4070, llama.cpp with the MTP pull request is the better answer.

If you have a 4080 or better, vLLM is super meta. (im running on 5090 its butter).

A QUICK PRIMER ON KV CACHE AND PAGED ATTENTION

When you generate text with a transformer, the model keeps a running cache of all previously computed attention keys and values. That cache grows linearly with context length and dominates memory at long contexts. (read this again, how all LLMs [ChatGPT] work).

QUANTIZATION QUALITY: WHICH FORMAT SHOULD YOU ACTUALLY RUN?

vLLM uses a different quantization ecosystem than llama.cpp. There is no GGUF here. The formats that matter in vLLM 0.9+ are FP8, AWQ, GPTQ, and INT4.

FP8 is the new production default. Quality is indistinguishable from BF16 in blind testing, and the speed and memory advantages are real on Hopper, Ada Lovelace, and MI300.

AWQ 4-bit is the practical sweet spot for 24GB consumer cards. Detectable but minor quality regression compared to FP16, with significant memory savings and excellent vLLM kernel support.

GPTQ INT4 is similar to AWQ in size and quality but has slightly slower kernel performance in current vLLM builds.

INT4 group-quantized formats lower than AWQ should be treated as experimental. The intelligence loss compounds badly in agent loops, where each step’s small error feeds into the next step’s prompt.

The general rule: if you are on an H100 or newer, run FP8 and stop thinking about it. If you are on a 3090 or 4090, run AWQ 4-bit unless you specifically need the last 1 to 2 points of benchmark quality.

HOW TO ACTUALLY RUN THIS TODAY

There are three things you need.

First, a model with MTP layers preserved. You want safetensors, not GGUF. Look for releases on Hugging Face tagged with “MTP” or “Native MTP Preserved.” Qwen has published official MTP-enabled checkpoints for the 3.6 series, and Unsloth has community maintained mirrors with additional fine tunes. I prefer HuggingFace.

Second, vLLM 0.9 or newer. Install with pip in a fresh Python 3.11 or 3.12 virtual environment. On AMD, install the ROCm wheel via Lemonade, which handles the dependency dance for you.

Third, the right server flags. A working command for a 24GB card looks like this:

vllm serve Qwen/Qwen3.6-27B-MTP \

quantization awq \

speculative-config ‘{”method”:”mtp”,”num_speculative_tokens”:3}’ \

max-model-len 65536 \

kv-cache-dtype fp8 \

gpu-memory-utilization 0.92 \

enable-prefix-caching

That gives you an OpenAI compatible endpoint on port 8000, native MTP with depth 3, FP8 KV cache, 64K context (even 32K should be fine for just 1 person), and prefix caching for agent loops (holy grail).

For multi-GPU setups, add tensor-parallel-size 2 (or 4, or 8) to scale across cards (I have yet to test this, but I will one day). For AMD, swap the awq quantization for fp8 or the rocm native quant of your choice.

DOES MTP WORK WITH OTHER ENGINES?

This is the second most common search question, so the short answer for each:

llama.cpp has native MTP via a recent pull request, not yet merged to main as of this writing. It is the right answer for single user inference, Mac users, and anyone on a 12GB or smaller GPU.

ExLlamaV3 does not yet ship native MTP. Its speculative decoding implementation works with a separate draft model.

MLX (Apple Silicon native) has MTP support in nightly builds, with performance still trailing llama.cpp on Metal. For Mac users today, llama.cpp is the right choice (I think Mac should go out of business personally, just expert fumblers).

Ollama and LM Studio are wrappers around llama.cpp and will gain MTP support once it merges upstream. Neither runs on vLLM.

For production serving, multi-user workloads (aka GigaChads), multi GPU deployments, or anything that needs throughput rather than just single stream speed, vLLM is the clear answer. Which means, its also the answer for enterprises; so, mastering it now will apply directly to job experience at IBM or NVIDIA.

WHAT THIS CHANGES

The honest answer is that hosting your own frontier class open weight model has been a promise more than a reality for the last year. The models were smart enough. The serving stack was too slow per dollar.

At 135 tokens per second on a single 3090 with batched throughput in the thousands on a dual 3090 rig, the math changed. Workloads that previously required cloud inference at frontier model prices now run on hardware that pays for itself in three to six months against a typical Claude or GPT subscription bill.

This is the moment to take self hosted inference seriously again if you had written it off after the last benchmark cycle.

The MTP work is moving fast, vLLM is shipping native support for new models almost weekly, and the cost curve has finally crossed the line where on device beats the API for the workloads most teams actually run.

If you are building anything that depends on cheap, private, high throughput inference, vLLM with MTP is the stack worth learning today...

What do you think?

Subscribe now

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

So much still to unpack just from the agent conference, let alone trying to keep up with the lightspeed pace of AI running on more affordable hardware. (Unrealized local potential from models like Gemma 4b, Qwen 27b..)

But one of my most memorable encounters from the conference, (read this letter if you are not up to speed: [So I Went to Manhattan for the AI Agent Conference [2026]) was with a 30 year dev that now works for Coder.com and sold me at the booth. Keep in mind, anyone that walks around is getting sold by the workers at the event; of which, coder.com was one. This guy was kind of just like…no need to sell. Which caught my attention.

Come to find out, he was one of the original developers behind the initial release of Xbox Live itself via Kubernetes.

He told me that GitLab and Coder.com are in real talks to merge…which is why I am now writing this letter to you; to explain how this marriage could end a hallmark like GitHub.

Share

What he described, when I pushed, was not a product roadmap. It was a structural argument. The forge GitHub built its empire on is an authentication and collaboration layer for human contributors. Every assumption it ships with, from rate limits to permission scopes to seat based pricing to the very shape of the pull request, encodes the constraint that contributors are humans typing at human speed.

The instant that constraint breaks, and it is breaking now, the forge stops being an asset and starts being a liability. The replacement, he said, was going to be self hosted source control fused with self hosted, policy bound development environments. In plain English: GitLab plus Coder (which…does not yet exist).

I went home and pulled on the thread. The more I read, the harder it was to argue with him. This is what I think is happening…and why I think the window for incumbents to respond is shorter than everyone is acting like it is:

THE REAL PROBLEM IS NOT VOLUME, IT IS PRINCIPALS

The easy version of this argument, the one you hear at every agent panel, is that agents commit more code than humans so the forges will get overwhelmed. That framing is correct but shallow. The volume problem is real and fixable. The deeper problem, the one that is not fixable inside the existing architecture, is that agents are not high speed humans. They are a different category of actor entirely.

GitHub’s identity model has one kind of citizen: a user, with an email, a 2FA device, and OAuth scopes. Everything else, deploy keys, fine grained tokens, GitHub Apps, the coding agent itself, is a hack bolted onto that single primitive. There is no first class concept of a non human principal that can be born, given a narrow mandate, watched while it works, killed, and have its actions automatically attributed back to the human who authorized it. The platform was not built with that abstraction because in 2008 there was no need for it.

In 2026, it is the only abstraction that matters.

An agent loose on your codebase is closer in kind to a contractor with elevated privileges than to a faster developer. It needs a scoped identity, a sandboxed environment, an explicit task, an expiration time, an audit trail, and a chain of accountability back to a human. None of those exist as first class objects in GitHub. They exist informally, glued together with personal access tokens and webhook secrets, and that glue is exactly where every credible piece of offensive security research in the last twelve months has found purchase.

This is why the architectural fight is happening below the application layer and not inside it. You cannot bolt non human principals onto a platform that was designed around the assumption of a human at every endpoint. You can only build the abstraction from scratch, or you can buy a platform that already has it. GitLab’s single application model and Coder’s workspace primitive, taken together, are the only combination on the market right now that gives an enterprise both halves.

WHY GITHUB CANNOT JUST FIX THIS

The standard rebuttal here is that Microsoft is the most resourced software company on earth and if there is a gap they will close it. They will ship features, yes. They have already shipped some. But the structural reasons they will not close the gap on the timeline that matters are worth being explicit about, because most analysts are still treating Microsoft’s response as a foregone conclusion.

First, GitHub is six products federated under one brand. GitHub.com, Actions, Packages, Codespaces, Advanced Security, and increasingly Azure for the parts that GitHub does not cover. The product was assembled through acquisition and organic growth, not designed.

Second, GitHub’s revenue model is structurally hostile to the agent era. Seat pricing assumes a roughly fixed ratio of contributors to value generated. The instant one developer is dispatching ten agents, that ratio breaks. Microsoft has two options and both are bad. Charge per agent and watch enterprises slow walk adoption to manage cost. Eat the cost and watch margin compress at the exact moment Copilot is already squeezing margin from the other side. GitLab’s pricing, which is built around CI minutes and compute rather than seats, sits much more comfortably in a world where most contributors are code, not people.

Third, Copilot is undermining the bundle from the inside. Every quarter Copilot wins on its own, the customer learns that the model vendor and the forge vendor can be separated. That lesson, once internalized at the CTO level, generalizes. If the model is detachable, why is the IDE not detachable? If the IDE is detachable, why is the CI runner not detachable? If the CI runner is detachable, what exactly am I paying GitHub Enterprise for that I could not get from a forge whose entire pitch is that I own the pieces? Microsoft is winning the model fight, briefly, at the cost of teaching its customers how to unbundle the platform underneath the model.

Fourth, the on premise version of GitHub, GitHub Enterprise Server, lags the cloud product by months and is treated internally as a tax rather than a strategy. The customers who care most about agent governance are exactly the customers who cannot use GitHub.com as it is built today. Microsoft would have to invert itself, become an on prem (local) first company for its most lucrative segment, to win that fight head on. They will not. They will ship VPC variants and dedicated tenancy and call it sovereign. The customers who matter will see the difference.

WHAT GITLAB ACTUALLY IS

GitLab is the boring incumbent whose obituary the industry has rewritten every two years and kept being wrong about. Here is what is actually true about it.

It is a single application. One codebase. One database. One permission model. One audit log. Inside that single application you get source control, code review, CI, package registry, container registry, security and dependency scanning, secrets detection, issues, planning, wiki, deployment, and an AI assistant that talks to all of the above. You can self host the whole thing with one Helm chart. You can air gap it. You can run it in a SCIF. You can deploy it on a submarine if you have the right people and a bad enough afternoon.

In the human era this was an acquired taste. Buyers tolerated GitHub’s federation because nobody was forcing them to write the unified governance query.

Subscribe now

In the agent era, the unified governance query is the product. Every regulator currently drafting a rule about AI in production code, and there are several, is going to demand a version of it. The platform that can answer it with one SQL statement against one database is going to look fundamentally different from the platform that needs to compose six APIs and reconcile three identity systems to produce the same answer.

This is why the boring incumbent is suddenly interesting. The thing GitLab has been right about for ten years, and largely punished for in the public markets, is the thing the next ten years is going to reward.

WHAT CODER ACTUALLY IS, AND WHY THE CODESPACES COMPARISON MISLEADS

Coder gets compared to Codespaces or Gitpod, and the comparison is shaped right but pointed in the wrong direction. Codespaces is a SaaS workspace. You click a button, Microsoft hands you a VM in their cloud, you write code, they bill you. The compute is theirs, the boundary is theirs, the policies are theirs.

Coder is not a workspace. Coder is a control plane. You install it on infrastructure you already own, Kubernetes or VMware or bare metal or AWS or whatever sovereign cloud your jurisdiction requires, and you define a workspace as a Terraform template. Base image, resource limits, network policies, egress allowlist, secret injection, identity binding, lifecycle.

When a developer or agent needs a workspace, Coder provisions one from your template, on your infrastructure, under your policies, with your observability hooks attached. When the task ends, the workspace evaporates and the credentials it held expire with it.

If you read those previous two paragraphs carefully you will notice it has almost nothing to do with developer experience and almost everything to do with how you would build an isolation primitive for a non human principal. That is the point. The agent era is not primarily a DX problem. It is a blast radius problem. Whoever ships the cleanest answer to “where does an agent live, what can it touch, how do I watch it, and how do I kill it” wins the layer of the stack that every other layer is going to depend on.

WHAT THE DEAL MUST CONTAIN TO MATTER

He did not tell me a deal was signed. He told me the shape of one. Here is what a serious GitLab and Coder partnership would have to ship, beyond co marketing, to actually move the market.

A workspace primitive native to the GitLab project. Open a repository, click a button, get an isolated, policy bound environment with the repo cloned, dependencies installed, secrets injected, and egress restricted. The template lives in the GitLab repository alongside the code. Platform teams own it. No drift, no bootstrap script that the agent has to negotiate from scratch, no surprise dependency that the agent installs because a model decided it was a good idea.

Federated identity with short lived, narrowly scoped tokens. The workspace gets a GitLab token bound to one project, one branch namespace, and a lifetime measured in minutes. When the workspace dies, the token is already invalid. A prompt injected agent that tries to pivot to another repository, exfiltrate to an external host, or persist credentials finds itself holding a token that does not authorize anything and a network that does not route.

Unified audit at the workspace identity level. Every file read, every command run, every commit pushed, every merge request opened, every pipeline triggered lands in the same audit log under the same identity as the workspace that produced it. The CISO query becomes a one liner. The compliance officer gets a real answer instead of an apology.

Cost attribution on infrastructure the customer already owns. The bill is a line item on a Kubernetes cluster the company already pays for, not a surprise SaaS invoice that scales with however many agents your engineering org decided to spawn this quarter. At fleet scale this is the difference between a sustainable platform and a budget incident.

That is the floor. The ceiling, if both companies execute, is an operating system for agent driven development. One place where you declare what agents are allowed to do, where you watch them do it, where their work is reviewed, and where the work that survives review becomes a deployment.

THE REGULATORY TAILWIND NOBODY IS PRICING IN

The reason I think this lands faster than the consensus expects is that the regulatory environment is about to do most of the selling for them, and almost no commentary I have read is taking that seriously yet.

The EU AI Act’s obligations for high risk systems came into force this year, and “code generation systems that materially affect production deployments in regulated sectors” is going to fall under high risk in every reasonable interpretation.

NIS2 requires demonstrable supply chain control for critical sectors across the EU and the deadline has passed. DORA, in financial services, demands operational resilience that explicitly extends to your software delivery pipeline.

In the US, federal procurement is quietly moving toward attestation requirements that no public SaaS forge can satisfy without operating a separate, lagging variant of its own product. Across regulated industries in roughly every G20 jurisdiction, the legal answer to “where does your code live and who can see it” stopped being “in the cloud, who cares” sometime in the last eighteen months and most procurement teams have not caught up.

The buyer for the forge has quietly changed. Five years ago the VP of engineering picked the forge and the CISO was a stakeholder. Today the CISO is the picker, and the VP of engineering is the stakeholder, and the CFO is auditing the agent line item every quarter because it is the fastest growing one. Once that is the buying committee, “single application, self hosted, single audit log, workspace level isolation, cost attributable to infrastructure we already own” is not a feature list. It is a procurement requirement.

This is the part Microsoft cannot beat with a better model. The model is not what the new buyer is buying.

THE FORCING FUNCTION

The reason the Coder dev said this was coming faster than people think is that the catalyst is not a product launch. It is an incident. The first time a publicly traded company has to disclose that a prompt injected agent inside a SaaS forge exfiltrated source code, pivoted to production, opened a backdoor merge request that got auto approved, or accepted a poisoned dependency that the model had no incentive to scrutinize, the agent governance conversation stops being a thought experiment at every other publicly traded company in the same week.

I do not know when that incident happens. Neither does anyone else. What I do know is that the surface area is growing exponentially, the offensive research is public, the median agent in production today is running with permissions a security review in 2019 would have rejected on sight, and the base rate for “this kind of incident occurs within eighteen months of conditions like these” is, historically, very high. Pick your favorite supply chain compromise of the last five years. The version of it that uses an agent as the delivery vehicle is shorter, cheaper, and more deniable.

When the incident lands, the companies that already have an answer for “agents run in Coder workspaces under GitLab policy on infrastructure we own” win the next two budget cycles of enterprise procurement. The companies that do not are going to spend those budget cycles writing memos explaining why they did not move sooner.

WHAT 2028 LOOKS LIKE IF THIS LANDS

Open source stays on GitHub. The network effect on public code is genuinely unbreakable on any reasonable horizon. Linus does not move the kernel. The npm ecosystem does not move. Hugging Face does not move. Your weekend project does not move. GitHub.com remains the global town square for public code and Microsoft remains a trillion dollar company that earned its place. Nothing in this thesis predicts otherwise.

The enterprise bifurcates. Banks, hospitals, defense, telcos, large law, pharma, and the upper tier of every regulated industry quietly migrate to a self hosted GitLab plus Coder stack and decommission GitHub Enterprise, Codespaces, hosted Actions runners, and Advanced Security in the same wave. The pitch writes itself: vendor consolidation, cost reduction, governance gain, regulatory defensibility. The first three reference customers are the hardest. After that it is a stampede, because in regulated industries nobody wants to be the company that did not move when their peers did.

The workspace becomes load bearing the way the container did after 2014. Today most teams do not have a name for the layer between “my laptop” and “production.” By 2028 they do, and the people who own that primitive own the workflow that runs on top of it.

GitLab’s AI story stops being an apology and starts being the product. Duo today is a Copilot chaser. Duo in a world where GitLab is the system of record for every agent action across the enterprise becomes something Copilot structurally cannot be, because Copilot does not have the unified data model underneath it. The model quality fight is the wrong axis. The right axis is “whose system can answer the questions a CISO actually has to ask,” and on that axis GitLab is positioned and GitHub is not.

The job called developer changes shape. Not “describe intent in plain English and a magic robot ships your product.” That fantasy will keep selling tickets at conferences and keep being wrong in practice. The actual shape is closer to fleet operator. You dispatch agents into governed workspaces, you review the diffs that survive policy, you merge what passes review, you deploy what ships. The IDE becomes a review surface. The forge becomes an orchestration layer. The number of people called developers does not shrink as much as the doomers think. The work each one does changes entirely.

WHAT TO WATCH OVER THE NEXT TWELVE MONTHS

Three signals.

One. A real GitLab and Coder integration. Not a co marketing webinar. Not a logo on a slide. A product. Templated workspaces tied to GitLab projects with token federation, policy enforcement, and unified audit. If that ships in 2026, the thesis is live and you should be paying close attention.

Two. A regulated industry reference customer. A name brand bank, a top ten hospital system, a major insurance carrier, a defense contractor publicly moving off GitHub Enterprise onto a self hosted GitLab plus Coder stack and citing agent governance as the reason. The first one is the hardest to land. The second one is the stampede.

Three. GitHub’s response. Specifically, whether Microsoft ships a true on premise, agent first, single application product, or whether they keep bolting features onto GitHub.com. The former means the fight is on. The latter means they are losing a segment without realizing they are losing it, which is how incumbents usually lose segments.

The forge was not built for this. Somebody is about to build the one that is. The person I met at the conference believes it is going to be them and GitLab, together.

After researching and building myself, I am inclined to believe him (i have 0 shares of anything coder or gitlab whatsoever, i get nothing out of sharing this)…

What do you think?

Subscribe now

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

Subscribe now

The implications of the conference were actually more valuable than the information presented there.

What the NYC AI Agent Conference Missed

I just spent three days in Manhattan at the ai agent conference [AI AGENT CONFERENCE 2026] and I left with a quiet, persistent feeling that the room was talking around the actual problem. The energy was real. The room was full of people building serious things. But almost no one in the conversations I sat through ($100B CEO’s talking to WSJ/CNN etc) was talking about NemoClaw, or about running enterprise-grade open weight models locally at all. Three other conversations dominated instead, and I want to lay each of them out before explaining why I think they share a blind spot.

The first conversation was about post-production agents and a redefinition of trust. The framing went like this: we can show you the agent’s reasoning, we can show you the data the agent saw, but the next step, the agent actually taking the action the data implies, is the step where no one can yet say with certainty that you can trust it. People argued, sincerely, that the very definition of trust would need to be revised to fit this new world. The proposed answer was almost always observability: better dashboards, better attribution, better post hoc explanations of what the agent did and why.

The second conversation was about agents as the new operating system. Every framework on stage seemed to want to be the substrate underneath every other agent. The runtime. The memory. The routing layer. The thing that other agents would, of course, be built on top of. The pitch was infrastructural. The implication was that whoever owned the runtime would own the next decade.

The third conversation was about layer capture. Orchestration. Memory stores. Inference gateways. MCP routers. Evaluation. Each company wanted to own one slab of the agent stack and be the default name in that slab. The theory of victory was horizontal: dominate one layer and let everyone above and below you depend on it.

What none of these three conversations questioned, and what I think is the actual blind spot, is the assumption sitting underneath all three of them. They all quietly assume that the enterprise customer will, of course, send their data to someone else’s model. Once you accept that assumption, trust becomes a problem you solve by adding observability after the fact. Operating system becomes a land grab for whose servers the data lands on. Layer capture becomes a question of who gets paid per token.

The harder, less glamorous, and ultimately more valuable problem is the one almost no one was working on out loud. It is: take someone with a real working knowledge of AI, sit them with a specific business until they understand the bottlenecks and the bottom line, and then translate that company’s actual workflow into a sandboxed set of tools the agent executes against. (I did see one CEO who understood this but he looked tired trying to explain it to everyone). The interesting question is not which model. The interesting question is how the agent receives data and how it executes tasks. That is the gold mine.

Niche dominance built on that translation work beats horizontal layer capture, because it produces something the enterprise can actually trust without first being asked to give up its data.

Let me show what that looks like in code, because I would rather quote the file than describe it. The first thing to notice is that, in this system, every tool the model can call goes through a single helper that reaches exactly one private API and nothing else.

import { McpServer } from “@modelcontextprotocol/sdk/server/mcp.js”;

import { StdioServerTransport } from “@modelcontextprotocol/sdk/server/stdio.js”;

import { z } from “zod”;

const BASE = process.env._BASE_URL || “https://.com”;

const API_KEY = process.env._API_KEY || “”;

const server = new McpServer({ name: “api”, version: “.0.0” });

async function fetchJSON(path, timeout = 000) {

const headers = {};

if (API_KEY) headers[”api”] = API_;

let res;

try {

res = await fetch(`${BASE}${path}`, {

headers, signal: AbortSignal.timeout(timeout)

});

} catch (err) {

const msg = err.name === “TimeoutError”

? “Request timed out.”

: `Network error: ${err.message}`;

return { error: msg };

}

if (!res.ok) {

const body = await res.text().catch(() => “”);

return { error: `API ${res.status}: ${body.slice(0, 00)}` };

}

try { return { data: await res.json() }; }

catch { return { error: “Failed to parse JSON.” }; }

}

There is no third party inference endpoint anywhere in this code path. The egress list is enforced at the sandbox boundary, not by polite intention. That one architectural decision is the precondition for every claim of trust the conference was reaching for.

The second thing worth showing is what the model is actually handed when a tool returns. In a lot of stacks I saw discussed, the model is asked to do arithmetic on raw JSON. Spreadsheets get dumped into prompts and the model is expected to do long division through a thirty-billion-parameter reasoner. The MCP server below does something different. It pre computes the verdict, in plain English, and the model receives the conclusion alongside the supporting numbers.

function pacingAnalysis(c, summary) {

const budget = Number(c.budget) || 0;

const spend = Number(c.spend ?? c.totalSpend) || 0;

const remaining = summary?.budgetRemaining

?? Math.max(0, budget - spend);

const daysLeft = summary?.daysRemaining ?? calcDaysLeft(c);

const dailyNeed = summary?.dailySpendNeeded

?? (daysLeft > 0 ? remaining / daysLeft : 0);

if (!budget) return “No budget data available.”;

if (daysLeft === 0) return `FLIGHT ENDED. Final spend: ${spend} of ${budget}.`;

const ySpend = Number(summary?.yesterdaySpend ?? 0);

const avgDaily = calcAvgDaily(c, spend);

const dailyActual = ySpend > 0 ? ySpend : avgDaily;

let verdict;

if (dailyActual >= dailyNeed * 0.95)

verdict = “ON PACE - no changes needed”;

else if (dailyActual >= dailyNeed * 0.70)

verdict = “SLIGHTLY BEHIND - monitor closely”;

else

verdict = “UNDERPACING - needs intervention”;

return [

verdict,

`Yesterday: ${dailyActual}/day vs ${dailyNeed}/day needed`,

`${daysLeft} days left, ${remaining} remaining of ${budget}`,

].join(”\n “);

}

The model is not being asked to be a calculator. It is being asked to do what language models are good at, which is produce a clear, well worded explanation of a verdict that has already been computed deterministically. This is what makes a local, open weight model competitive for real work. The arithmetic is not its job, and we do not pretend that it is.

The third excerpt is the one I think the conference would have benefited from most. It is not impressive. It is not on a roadmap slide. It is the watchdog that runs inside the same sandbox and quietly fixes things before the agent or the human ever has to notice.

async function triggerPlatformRefresh(platform) {

const headers = {};

if (API_KEY) headers[”api-key”] = _API_KEY;

const url = `${_BASE_URL}/api/refresh/cron/platform/${platform}`;

const res = await httpFetch(url, { headers, timeout: HEAL_TIMEOUT });

if (!res.ok) throw new Error(`${res.status} response`);

return res.data;

}

for (const platform of stalePlatforms) {

const name = platform.platform;

const entry = state.stalePlatforms[name] || { healAttempts: 0 };

if (entry.healAttempts >= 3) {

failed.push({ ...platform, entry });

continue;

}

try {

await triggerPlatformRefresh(name);

healed.push(name);

delete state.stalePlatforms[name];

} catch (err) {

entry.healAttempts = (entry.healAttempts || 0) + 1;

state.stalePlatforms[name] = entry;

}

}

The point of this excerpt is not the code. The point is that the agent is not the smart part of the system. The boring infrastructure around the agent, the watchdog, the retry budget, the alert cooldown, the structured failure path, campaign study loop, access to proprietary software API, is what makes the whole thing trustworthy at scale. The model is consulted. The system runs.

Step back from the three excerpts, and the conference’s central question answers itself...

Trust is not an observability layer that gets bolted on after the fact. Trust is what you get when the data physically cannot leave the building, when the egress list is two destinations long, and when the model is one component inside a sandbox the operator owns and can audit line by line. The redefinition of trust the conference was groping toward is not a product category. It is an implementation problem. It has, quietly, already been solved (NemoClaw) by anyone who chose to take hold and build.

The cost of this approach is not the eye watering number of people associated with running models on-prem. The footprint is one workstation, one open weight model, and one carefully written sandbox policy (YAML privacy router in NemoClaw shell for example). There are no per token bills. There are no rate limits. There is no third party data exposure. Frontier models are extraordinary research artifacts, but they are not the only way to put real intelligence into a real business.

The honest summary of the room at the conference is that no one yet knows the right answer. That is fine. What is striking is that almost no one is building the unglamorous, obvious moat. The opportunity for us is sitting there in the gap between a clean conference talk about agent trust and a directory of code that actually delivers it.

I would rather spend the next year building inside that gap than spend it on stage talking about how it might one day be closed.

What do you think?

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

The setup is simple to describe and hard to believe until you see it working.

Share

You deploy one AI agent into your company Slack. Your team talks to it like a coworker. They ask it questions, give it tasks, loop it into channels. It responds in threads. It remembers context.

What your team does not see is what is happening underneath.

That one Slack agent is the front door to a network of hundreds, potentially thousands, of specialized agents running on hardware sitting in your office or your server closet. Agents that handle procurement, reconcile invoices, monitor infrastructure, draft contracts, triage support tickets, update your CRM, pull reports from proprietary systems your company built ten years ago that nobody else’s SaaS can touch.

One interface. One conversation. An army behind it.

This is not theoretical. This is NemoClaw and OpenClaw running right now on real hardware in real companies (I am doing it myself [only been out 1 month] on full-stack projects I created).

THE LEVERAGE OF NEMOCLAW ARCHITECTURE

If you read my Finding NemoClaw piece, {Finding NemoClaw? [2026]} you already know the origin story. Adobe, Salesforce, SAP, ServiceNow, Siemens, CrowdStrike, Atlassian, Cadence, Synopsys, Palantir, and Cisco were named as building on the Agent Toolkit the day this was announced 1 month ago. These companies are not experimenting. They are deploying.

Here is what you need to understand about the architecture, because this is where the real leverage lives.

NemoClaw wraps OpenClaw in three layers of control that make this viable for any company that cares about its own data:

A kernel-level sandbox. Every agent runs inside an isolated Linux (finally) environment. Landlock filesystem restrictions, seccomp syscall filtering, network namespace isolation, capability dropping. Deny-by-default on everything. The agent cannot access files, networks, or APIs unless you explicitly approve them through policy. This is not a guardrail bolted on after the fact. It is baked into the operating layer itself.

An out-of-process policy engine. This is the part that matters for enterprise. The policy engine runs outside the agent. A compromised or hallucinating agent cannot override its own restrictions. You define what each agent can touch using YAML policy files, and the enforcement sits in a place the agent literally cannot reach. You can view the live enforced policy at any time with a single command.

A privacy router. This is the piece that makes the whole thing real for companies with sensitive data. The privacy router inspects every query and classifies its sensitivity. Anything containing PII, financial data, proprietary code, or internal documentation gets routed to your local Nemotron model. It never leaves your building. Non-sensitive queries can optionally route to cloud models for speed (i wouldn’t unless needed).

That last point alone eliminates about 80% of the objections legal teams have been throwing at AI deployments for the past two years.

THE SLACK AGENT IS THE FRONT DOOR, NOT THE BRAIN

Here is where most people get the architecture wrong when they first hear about it.

They think “Slack bot” and picture some command-line toy that responds to slash commands with canned answers. That is not what this is.

The Slack agent is the conversational interface to your entire agent infrastructure. OpenClaw’s Channel layer natively supports over 23 messaging platforms: Slack, Teams, Discord, WhatsApp, Telegram, Signal, Google Chat, Matrix, LINE, Mattermost, and more. Slack is the one that matters for most companies because that is where your team already lives. But the point is this works wherever your people already communicate.

Behind that one Slack connection, OpenClaw’s orchestration layer manages a fleet of specialized agents. Each one has its own sandbox, its own permissions, its own tool access, its own workspace. When someone in your sales channel asks the bot “what is the status of the Acme renewal and can you draft the follow-up email based on the last three calls,” that request fans out to multiple specialized agents, each with scoped access to only what they need, each in their own isolated sandbox. Your salesperson sees one reply in a Slack thread. Multiple agents did the work behind the wall.

And here is where it gets powerful. You are not managing these agents through some web dashboard you have to learn. You manage them through the CLI, which means everything is scriptable, auditable, and version-controllable.

THE CLI IS THE REAL SUPERPOWER

This is the section the technical founders reading this have been waiting for.

OpenClaw was designed from the ground up with the command line as its primary entry point. This is not a GUI product with a CLI bolted on. The terminal is the control surface, and it is extensive.

Here is what you can do without ever leaving your terminal:

Agent fleet management. Create, inspect, delete, and configure isolated agents. Each agent gets its own workspace, authentication, model configuration, and channel bindings. Spin up a new specialized agent in seconds:

openclaw agents add procurement-bot

Bind it to a specific Slack channel so it only responds where it is relevant:

openclaw agents bind --agent procurement-bot --channel #purchasing

Inference control. Set which models each agent uses. Configure fallback chains so if your local Nemotron is at capacity, queries cascade to your backup. Scan for available models across all providers in one shot:

openclaw models list --all

openclaw models fallbacks add gpt-4o

openclaw models set nemotron-3-super

Scheduling and automation. Set up cron jobs that let your agents run tasks on a schedule. Daily reports, weekly reconciliations, hourly monitoring checks:

openclaw cron add --name “daily-inventory” --every “24h” --message “Run full inventory reconciliation and post summary to #ops”

Your procurement agent can audit vendor invoices every morning before anyone clocks in. Your infrastructure agent can run health checks every hour and only ping the team channel if something is wrong.

Browser automation. This one surprises people. OpenClaw has a built-in browser control layer. Navigate pages, fill forms, take screenshots, capture page structure, simulate clicks and keyboard input. Your agents can interact with web-based tools that do not have APIs (which is insane):

openclaw browser navigate “https://internal-erp.company.com/reports”

openclaw browser screenshot

openclaw browser fill ‘{”vendor”: “Acme”, “date_range”: “Q1 2026”}’

That legacy ERP system with no API that someone on your team logs into manually every day to pull numbers? An agent can do that now.

Node management. Pair additional hardware as compute nodes. Approve or reject pairing requests. Invoke remote commands. Your server closet becomes a distributed agent compute cluster:

openclaw nodes list

openclaw nodes approve <id>

openclaw nodes invoke --node <id> --command “openclaw agents list”

On the NemoClaw side, the control surface is just as deep:

nemoclaw list # see all your sandboxes

nemoclaw ops-agent connect # drop into a sandbox shell

nemoclaw ops-agent status # health, inference config

nemoclaw ops-agent logs --follow # stream logs in real time

nemoclaw ops-agent policy-add slack # apply Slack network preset

nemoclaw ops-agent policy-list # see active and available presets

nemoclaw ops-agent snapshot create # backup agent state

nemoclaw ops-agent snapshot restore # rollback if needed

nemoclaw ops-agent skill install ./custom-skill # deploy new capability

nemoclaw backup-all # backup every running sandbox

nemoclaw debug --sandbox ops-agent # collect diagnostics

openshell term # monitor and approve network requests live

Snapshots and backups. Create timestamped state backups of any agent, restore to any previous snapshot, and backup all running sandboxes with one command. This is the kind of operational maturity that takes most platforms years to develop.

Everything here is scriptable. Pipe it into your CI/CD. Version control your policy files. Automate your agent deployments the same way you automate your infrastructure. This is infrastructure as code, applied to your entire AI agent fleet.

THE HARDWARE

One of the most underreported parts of this story is what it runs on.

For full local inference with Nemotron 3 Super 120B, you want a DGX Spark or equivalent. But here is what we are figuring out: you do not need the biggest model for every agent. The privacy router lets you run Nemotron Nano 4B for simple routing and triage tasks and only spin up the 120B for complex reasoning. Most of the agent fleet runs light. Dell Pro Max with GB10 gives you 128GB coherent unified memory. The GB300 variant delivers 748GB coherent memory and up to 20 petaflops in a desktop form factor.

The deployment is one command:

curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

An onboarding wizard walks you through sandbox creation, inference configuration, and security policies. Connect your Slack bot token. Approve the pairing. You are live.

The people who read my junk drawer piece are already ahead here: {Is Your Junk Drawer a Supercomputer? [2026]}

THE COMPOUNDING PROBLEM (FOR EVERYONE WHO WAITS)

This is the part I need you to really hear.

The companies deploying NemoClaw right now are not just saving money on SaaS subscriptions or headcount. They are building institutional muscle memory that compounds every single day.

Every time an agent processes a workflow, it gets context. Every integration adds capability. Every policy refinement makes the system smarter about what it should and should not touch. Every custom skill deployed, every cron job configured, every node paired adds surface area that no competitor can replicate because they do not have access to the same internal systems, the same institutional knowledge, the same proprietary data sitting behind the privacy router.

And here is the compounding math that should keep you up at night:

A company that deployed NemoClaw in March has had over a month of policy refinement, skill development, cron job tuning, and workflow optimization. Their agents have memory. Their team has developed intuition for what to delegate and what to keep human. Their snapshot history gives them rollback safety that lets them move faster.

A company that deploys in July starts from zero.

But the March company is not four months ahead. They are four months of compounding ahead. Their agents are better tuned to company-specific context. Their integrations are more mature. Their CLI scripts are battle-tested. That gap does not shrink over time. It widens…

This is the same dynamic that played out with companies that adopted cloud infrastructure early versus those that waited.

Gartner says 40% of enterprise apps will have task-specific AI agents embedded by end of year. Only 5% had them in 2025. That curve is steep, and the companies riding the front of it are building advantages that compound with every passing week.

The companies that will dominate the next decade are not the ones with the best AI models. The models are converging. Every benchmark is tightening. It is the companies with the deepest integration into their own proprietary systems, the most refined agent policies, the most mature orchestration layers, and the most battle-tested CLI automation that will be impossible to catch.

NemoClaw is how you build that. The Slack bot is how your team uses it without reading a single line of documentation. And the CLI is how you, the person building this, control every piece of it with the precision and speed that no dashboard will ever match.

The window where this feels like a novelty is closing fast. On the other side of that window, it is just how companies operate. And the ones who got there first will be very, very hard to catch.

Start Monday…

Subscribe now

What do you think?

God-Willing, see you at the next letter

GRACE & PEACE

VISIT JoeGuglielmucci.com TODAY

Last year, 300 people showed up. This year, over 2,000 are expected. Every sponsorship package is sold out. The speaker list reads like someone merged the org charts of Thomson Reuters, GitHub, OpenAI, Bloomberg,…

Read more

It wi…

Read more

This isn’t a hot take. It’s something you can watch happen in real time, twice, in the span of a single week.

Read more

Read more

Before we start, I wanted to deeply thank the newest paid subscriber to The Gug Letter, @ Chad Cummings! Greatly appreciate you!

Chad and I went to law school together. He is by far the smarter one between us & also writes an incredible Law Journal worth reading.

Really enjoyed this recent edition: Tax Consequences of Using a Single-Member LLC as …

Read more

Let me explain.

Read more

I didn’t cut any corners, which means you won…

Read more

DATA

Removing all trash signals, here is what the data says: …

Read more

---

Last August, I wrote that generative AI would bring software to $0: Will Generative AI Bring Software to $0?. The…

Read more

The silicon that powers ChatGPT is the same silicon that runs Call of Duty. That's not…

Read more

Today is the Nativity of Our Lord, Christ is Born! Happy New Year and Happy Holidays to everyone!

I am excited and blessed to build more in 2026.

Back to our advertised schedule of 2 free letters and 1 paid letter each week. Want to make sure each letter is value packed and not just ai slop for you guys.

Share

I thought it would be fittin…

Read more

Read more

…yes.

[Genesis of Pay Per Crawl [2025] - Read this first.

Read more

If you thought the 2008 mortgage crisis was the peak of financial alchemy, meet paper Bitcoin — the sleaziest trick in the 2025 playbook.

Paper Bitcoin isn’t real BTC. It’s a derivative claim — an ETF share, a futures contract, a perpetual swap, or an O…

Read more

I wanted to write an article the second this happened but I thought best to wait, sit back, actually study what was being “sold” as the cause and what might actually be the cause. Could also be a mix of both…but here is what I think:

On …

Read more