Measure $\alpha$, then decide whether to speculate

Setup

The textbook model of speculative decoding is closed-form; what it looks like in production is not — so we measured it, on real workloads and real silicon.

Speculative decoding trades cheap draft tokens for expensive target verifications. A small model proposes $\gamma$ tokens; the target model checks them in a single forward pass and keeps the longest correct prefix. The mechanism is exact — the output distribution is unchanged — so the only question is economic: how many drafted tokens survive, and at what cost. That survival rate is the acceptance rate, and it controls everything downstream.

Define per-position acceptance as one minus the total-variation distance between the draft distribution $q_t$ and the target distribution $p_t$ at position $t$:

Collapse the sequence to a single number under the i.i.d. approximation $\beta_t \approx \alpha$, and the expected number of tokens accepted per verification round — together with the resulting speedup — falls out in closed form:

Here $\alpha$ is the acceptance rate, $\gamma$ the draft length, and $c = t_{\text{draft}}/t_{\text{target}}$ the draft/target cost ratio. This is a clean, teachable model. It is also, in every term, a simplification: $\alpha$ is not a scalar, the tokens are not independent, and $c$ is not a constant. Each of those simplifications hides a place where the real economics diverge — sometimes by an order of magnitude. What follows is what we found when we stopped assuming and started measuring.¹

Acceptance isn't constant or independent

The single-$\alpha$ model treats every drafted position as an independent coin flip, which implies the probability of accepting $t$ tokens decays geometrically as $\alpha^{t}$. That is not what the data does. When we condition on position and on the draft having cleared the tokens before it, acceptance is sharply non-uniform.

The first token off the draft is genuinely hard — its unconditional acceptance sits around $0.4$. But conditional on clearing that first token, the draft and target have effectively agreed on a trajectory, and deeper positions accept at $0.9$–$1.0$. The relevant quantity is this conditional acceptance, not the unconditional $\alpha^{t}$ that the i.i.d. model decays. Once the draft is on the rails, it stays on them.

The operational consequence is that a fixed $\gamma$ is the wrong control. When the conditional rate is high, a longer draft is nearly free yield; when the first token stalls, a long draft is wasted compute. Adaptive $\gamma$ wins — extend the draft while conditional acceptance stays near one, and cut it the moment it drops.

Realized speedup varies $\sim$4$\times$ across workloads

Aggregating conditional acceptance into an effective $\alpha$ at $\gamma=6$, we saw it range from roughly $0.40$ on open-ended prose to roughly $0.97$ on structured code — a nearly order-of-magnitude swing in accepted-draft yield. That swing is floored, not amplified, by the fact that the target always emits at least one free token per round: even a useless draft cannot make you slower than the baseline in token count. The result is realized end-to-end speedups from $\sim$1.4$\times$ on creative prose to $\sim$6$\times$ on code — the same optimization, opposite P&L. This swing is plotted in Figure 1.

The takeaway is not that speculative decoding is fast. It is that a single flag delivers a $1.4\times$ or a $6\times$ depending entirely on what the request is — and you cannot know which without measuring the workload's acceptance curve.

Memory-bandwidth arbitrage, with a ceiling

Why does speculation help at all? Because low-batch decode is starved for arithmetic. A single-stream decode step does on the order of $1$–$2$ FLOP/byte of weights moved — orders of magnitude below the hardware's compute-to-bandwidth ridge. The accelerator spends its time waiting on memory, not computing. Speculative verification converts that idle compute into work: instead of a memory-bound GEMV per token, the target runs a compute-bound GEMM over $\gamma{+}1$ candidate positions at once. You are buying tokens with FLOPs you were already paying for.

But this is arbitrage against a fixed price, and the price has a floor. As batch size $B$ grows, ordinary decode already fills the pipes — the arithmetic intensity of a plain batched GEMV climbs toward the ridge on its own, and the free FLOPs speculation was exploiting disappear. The crossover, where $\gamma{+}1$ speculative positions stop being cheaper than simply batching more requests, scales as:

On an H100 in BF16 the ridge $I^{\ast}$ is roughly $295$ FLOP/byte, which puts $B^{\ast} \approx 49$ at $\gamma=5$.² In practice, production systems disable speculation well before that — around batch 32 — because past that point the concurrency is more valuable spent on real requests than on speculative candidates.

And the cost ratio $c$ is not a constant either. We measured it from roughly $0.02$ — a tiny draft colocated on the same device, nearly free — up to about $2$, where the draft is large enough that it does not fit alongside the target and every drafted token costs more than the target token it hopes to save. In that regime the closed-form speedup drops below one: speculation is a slowdown. The same equation that promises 6$\times$ promises a regression, and only the measured $c$ tells you which side of the line you are on.

Every term in the textbook speedup formula turned out to be a decision, not a given. Acceptance depends on the request; the crossover depends on the batch; the cost ratio depends on the draft you chose and the device it runs on. Inference optimization is therefore not a static flag you turn on but a per-request control policy: measure $\alpha$, choose the draft, choose $\gamma$, choose the accelerator, watch batch pressure, and disable when it stops paying.

The teams that win the inference cost curve are not the ones that turned on speculative decoding; they are the ones that measured $\alpha$, chose $\gamma$ per workload, and knew exactly when to stop. The formula was never the answer — it was the list of things worth measuring.