Measure $\alpha$, then decide whether to speculate
Turn on speculative decoding and you might get a $1.4\times$ speedup. Or a $6\times$. Same flag, same model — wildly different payoff. The difference comes down to one number almost nobody measures: the acceptance rate, $\alpha$.
Speculative decoding is one of the best-understood tricks in LLM inference. A small “draft” model proposes $\gamma$ tokens; the large “target” model checks them in a single forward pass and keeps the longest correct prefix. It is mathematically exact — the output distribution is unchanged — so the only question left is economic: how many drafted tokens survive, and at what cost? The textbook answer is closed-form:
Clean and teachable. It is also, in every term, a simplification: $\alpha$ is not a scalar, the tokens are not independent, and $c$ — the draft-to-target cost ratio — is not a constant. Each simplification hides a place where the real economics diverge, sometimes by an order of magnitude. So we stopped assuming and measured it, on real workloads and real silicon.
Every term in the textbook speedup formula turned out to be a decision, not a given.
Acceptance isn't constant — or independent
The single-$\alpha$ model treats every drafted position as an independent coin flip, so accepting $t$ tokens should decay geometrically as $\alpha^{t}$. The data does not behave that way. The first token off the draft is genuinely hard — its unconditional acceptance sits around $0.4$. But conditional on clearing that first token, the draft and target have effectively agreed on a trajectory, and deeper positions accept at $0.9$–$1.0$.
That single fact rewrites the control policy. A fixed $\gamma$ is the wrong knob. Adaptive $\gamma$ wins: extend the draft while conditional acceptance stays near one, and cut it the moment it drops.
Same optimization, opposite P&L
Collapse conditional acceptance into an effective $\alpha$ at $\gamma=6$, and it ranges from about $0.40$ on open-ended prose to $0.97$ on structured code — a $9.7\times$ swing in accepted-draft yield. The swing is floored, not amplified: the target always emits at least one free token per round, so even a useless draft cannot make you slower in token count. End to end, that lands as realized speedups from $1.4\times$ to $6\times$, entirely by workload:
Why it works at all — and where it stops
Speculation helps because low-batch decode is starved for arithmetic. A single-stream decode step does only $1$–$2$ FLOP per byte of weights moved — orders of magnitude below the hardware's compute-to-bandwidth ridge. Verification converts that idle compute into work: instead of a memory-bound operation per token, the target runs a compute-bound pass over $\gamma{+}1$ candidate positions at once. You are buying tokens with FLOPs you were already paying for.
But it is arbitrage against a fixed price, with a floor. As batch size grows, ordinary decode fills the pipes on its own and the free FLOPs vanish. The crossover scales as:
On an H100 in BF16 the ridge is about $295$ FLOP/byte, which puts the crossover near batch $49$ at $\gamma=5$. In practice, production systems disable speculation earlier — around batch 32 — because past that point concurrency is better spent on real requests than speculative candidates. And $c$ is not fixed either: we measured it from roughly $0.02$ (a tiny draft on the same device, nearly free) up to about $2$ (a draft so large that every drafted token costs more than the target token it hoped to save). At the high end, the closed-form speedup drops below $1$ — speculation becomes a slowdown.
The same equation that promises $6\times$ promises a regression, and only the measured $c$ tells you which side of the line you are on.
What we measured vs. what we cite
| Claim | Value | Source |
|---|---|---|
| Acceptance is non-i.i.d. | first token $\sim$0.4, deeper $\sim$0.9–1.0 | measured |
| Effective $\alpha$ ($\gamma=6$, greedy) | 0.40–0.97 | measured |
| Accepted-draft yield swing | 9.7$\times$ across workloads | measured |
| Draft/target cost ratio $c$ | 0.02–2 | measured |
| Batch-32 disable point | production / EAGLE-3 practice | cited |
| $B^{\ast} \approx \text{ridge}/(\gamma+1)$ | scaling heuristic, not an exact bound | cited |
| H100 ridge | $\sim$295 FLOP/byte (BF16), $\sim$591 (FP8) | vendor roofline |
The takeaway
Inference optimization is not a static flag you flip on. It is a per-request control policy: measure $\alpha$, choose the draft, choose $\gamma$, choose the accelerator, watch batch pressure, and disable when it stops paying. That is exactly the kind of decision Natum makes automatically — step by step, across models and accelerators.
The teams that win the inference cost curve are not the ones that turned on speculative decoding; they are the ones that measured $\alpha$, chose $\gamma$ per workload, and knew exactly when to stop. The formula was never the answer — it was the list of things worth measuring.