Autoregressive generation is a queue at a single counter. A large language model produces one token per forward pass, and each pass costs roughly the same whether it emits a comma or the keystone of an argument. The arithmetic is unkind: a model whose weights barely fit in memory pays, for every single token, the full price of dragging those weights past the data. The machine is rarely thinking in that moment. It is mostly waiting on bandwidth.

There is a second, older observation hiding underneath the first: checking an answer is cheaper than producing one, and checking is parallel in a way that production is not. A human can skim a proof faster than she can write it. A model can score a hundred candidate completions in one batched pass, but it cannot write a hundred completions in one pass.

Three literatures that rarely cite each other have converged on this asymmetry from three directions. Inference researchers built speculative decoding: a cheap drafter guesses several tokens, a heavy model verifies them in a single pass, and a rejection-sampling rule guarantees the output is exactly what the heavy model would have said. Multi-agent system researchers built LLM teams: small models draft, debate, and propose, while larger models arbitrate, aggregate, or break ties — the same division of labor wearing an org chart. And reinforcement-learning researchers built selective self-improvement: policies sample, a verifier scores, and gradients or data are released only where the policy currently fails — the newest variant literally names itself just-in-time.

This post walks the three literatures in that order, then argues they are one loop — speculate, verify, adapt — seen at three timescales: the token, the task, and the deployment. We will use one notation throughout and, at the end, one comparison table.

Contents

– Background: one loop, three timescales

– Notation

– Speculative decoding: the rejection-sampling contract

– Who drafts? The evolution of the cheap proposer

– Agents as speculation: teams, cascades, and token economics

– Don’t throw away the verdict: verification becomes training signal

– Where the loop strains

– References

Background: one loop, three timescales

The loop is short enough to fit in a sentence. A cheap process proposes; an expensive process checks; the verdict decides what survives — and, if the system is allowed to learn, what gets remembered or trained on.

At the token timescale this is speculative decoding: draft γ tokens greedily and cheaply, verify all of them in one forward pass of the target model, accept a prefix, resample the first disagreement. The expensive model’s judgment is exact, so the loop is provably lossless.

At the task timescale this is a multi-agent system: several models (or several copies of one model) draft answers or actions; a heavier model, a vote, or an aggregator verifies. Here verification is no longer exact — judges err, majorities can be confidently wrong — but the economics are the same shape: proposals are parallel and cheap per unit, arbitration is expensive and scarce.

At the deployment timescale this is continual adaptation: the policy acts, the environment or a verifier returns a verdict, and the system updates — sometimes weights, sometimes a memory, sometimes only the budget it grants itself next time. Just-in-time RL (JitRL), the newest member of this family, updates the logits at inference time from retrieved experience, without touching a single weight.

One caveat before the math. Everything below is a synthesis of public papers, and every number is as reported by its authors. When the literature disagrees with itself, we say so; when it is silent, we do not fill the gap with confidence.

Notation

Symbol	Meaning
x1:t = (x1, …, xt)	Token sequence and prefix
p_θ(· | x1:t)	Target (“verifier”) next-token distribution, parameters θ
q_φ(· | x1:t)	Drafter (“speculator”) next-token distribution, parameters φ
zt ∈ ℝ|𝒱|	Logits; p_θ(x | x1:t) = softmax(zt)_x
ht	Final-layer hidden state; ft a lower (feature-level) representation
γ	Number of tokens drafted per cycle
α	Expected per-token acceptance rate of the drafter
c = τq / τp	Drafter per-token cost relative to one target forward pass
G, ri, Âi	Group size, reward of sample i, and its (group-relative) advantage
π_θ(a | s)	Agent policy over actions given a state
ℳ, 𝒩(s)	Experience memory and the trajectories retrieved for state s
β	Step size or KL budget (context-dependent; always stated)

Speculative decoding: the rejection-sampling contract

The founding observation of the field is that the target model is its own perfect verifier. Verifying γ proposed tokens costs one forward pass — the same pass that would have generated a single token. If a cheap drafter can guess a few tokens that the target would have accepted anyway, each cycle emits multiple tokens for the price of one.

Two papers appeared within weeks of each other in late 2022 and early 2023 with this contract: Fast Inference from Transformers via Speculative Decoding [1] and Accelerating Large Language Model Decoding with Speculative Sampling [2]. The mechanics, in our notation:

1. Draft. Sample x_t+1, …, x_t+γ ~ q_φ, autoregressively, cheaply.
2. Verify. Run the target once on the extended prefix to obtain p_θ(· | x_1:t+j) for every draft position j — all in one pass.
3. Accept or resample. Going left to right, accept x_t+j with probability

At the first rejection, discard the rest of the draft and instead resample a replacement token from the residual distribution

which is the part of the target’s opinion the drafter under-rated. If every draft token is accepted, the bonus token the verification pass itself computes is emitted for free, so a cycle yields between 1 and γ + 1 tokens.

The magic of the residual rule is that it makes the whole scheme an exact rejection sampler for p_θ: the marginal distribution of every emitted token is precisely the target’s, token for token. No quality metric moves. This is why the authors could advertise it as lossless acceleration, and why the method spread through production stacks faster than most research does — [1] reported 2–3× speedups on T5-XXL, [2] reported 2–2.5× on a 70B Chinchilla model.

The arithmetic of the trade. Let α be the expected acceptance rate of an individual draft token and assume (as the founding papers do, for analysis) that acceptances are independent across the block. The expected number of tokens emitted per cycle is geometric:

where the +1 exponent quietly encodes the bonus token. A cycle costs γ drafter steps plus one target pass, so with c = τ_q/τ_p the speedup over vanilla decoding is

Everything the next decade of papers fight over is visible in this one expression. Raise α (draft better) and the numerator grows. Lower c (draft cheaper) and the denominator shrinks. Grow γ and you gain only as fast as α^γ+1 dies — a long draft behind a mediocre drafter is pure loss, which is why the field’s energy went into the drafter, not the schedule.

Who drafts? The evolution of the cheap proposer

Borrow a smaller brain

The obvious first move, taken by both founding papers, is a second, smaller model as the drafter. This works — the smaller model shares enough distributional mass with the target on easy tokens (articles, common code idioms, boilerplate) to sustain a healthy α — but it carries luggage: a second set of weights in memory, a vocabulary that must align token-for-token with the target’s, and a drift problem, since the drafter must be re-aligned whenever the target is updated. The search for a better drafter is the search to delete these costs one at a time.

Skip layers, keep the weights: self-speculation

Draft & Verify [3] noticed that the cheapest possible drafter is the target model with its middle layers amputated: draft with the early layers (plus an early-exit head), skip the middle, then verify with the full model. Nothing new is trained; nothing new lives in memory. LayerSkip [4] went further and made the model aware during training that it would be asked to exit early: training uses layer dropout (heavier on later layers) plus an early-exit loss shared across layers, so at inference the same weights can draft by exiting at layer k and verify by running to the end. Reported self-speculative speedups land in the modest but honest 1.3–1.7× range — with a drafter that costs zero extra memory.

Many heads on one neck: Medusa

Medusa [5] attacks the sequential bottleneck directly. Instead of one next-token head, the model grows a handful of extra decoding heads on the final hidden state h_t, each trained to predict the token at offset t+1, t+2, …, t+k simultaneously — the backbone stays frozen, only the heads train. Because each head’s top predictions form a small set of candidates per position, the candidates across positions weave a tree of possible continuations, and a tree attention mask lets the target verify the entire tree in a single forward pass, masking each candidate against only its own ancestors. Combined with a “typical acceptance” relaxation (accept tokens the target deems typical rather than argmax-probable, keeping the method near-lossless in practice and fully lossless in its strict variant), Medusa reported delay reductions of roughly 2–3.6× across models and tasks.

Drafting in feature space: EAGLE, twice over

The EAGLE line asked a sharper question: what should the drafter autoregress? Tokens are a poor medium — the token at position t+1 has no use for the token at t, only for the representation the target will build from it. EAGLE [6] drafts at the feature level (the second-to-top-layer hidden state), autoregressing f_t+1 from f_t with a lightweight head, then decoding to tokens — and crucially, at each step the verified target feature is fed back in, so the drafter never accumulates drift. Its other insight is that feature prediction is multi-modal: the drafter samples features rather than taking a point estimate, covering the target’s uncertainty. EAGLE reported 2.7–3.5× on chat models of the day.

EAGLE-2 [7] noticed that the draft tree was static — the same comb every cycle — while the useful part of the tree is intensely context-dependent (“3 + 4 =” has one obvious continuation; “the capital of” has dozens). It replaced the fixed tree with a dynamic draft tree: expand nodes by a confidence score computed from the target’s own verification distribution, allocating the draft budget where acceptance probability actually lives. Reported speedups rose to the 3–4× range. EAGLE-3 [8] then broke the last taboo — do not let the drafter see the target’s internals during training — with training-time test: the draft model is trained consuming the target model’s deep features, and multi-layer feature fusion replaces the single-layer input, closing most of the remaining train–inference gap. On very large models, reported speedups climb past 4× toward 6×.

Drafters with no training at all

If drafting is guessing what comes next, there are older oracles than neural networks. REST [9] keeps a datastore of (context, continuation) pairs and drafts by retrieving continuations for contexts similar to the current prefix — retrieval as a free drafter, effective precisely where text is repetitive (editing, summarization, code repair), with reported speedups around 1.6–2.4×. Lookahead decoding [10] needs neither a drafter nor a datastore: it generates a pool of n-grams in parallel and verifies them with Jacobi-style parallel fixed-point iterations, breaking the sequential dependency inside the decoding loop itself; reported speedups range from about 1.5× on open-ended chat to 4× where parallel compute is plentiful. Sitting nearby in the family tree, contrastive decoding [11] uses a weak “amateur” model not to draft but to veto: the objective subtracts the amateur’s logits from the expert’s, suppressing plausible-to-novices errors. It has no verification step and changes the decoding objective rather than preserving it, but it shares the core bet — a cheap second opinion reshapes the cheap proposals.

The systems view: trees meet hardware

Two systems papers close the inference arc by reminding us the loop runs on silicon. Sequoia [12] chooses the token tree itself by dynamic programming, matching tree shape to the hardware’s verification cost — and finds the optimal tree is workload- and hardware-specific, with the largest gains exactly where bandwidth hurts most: very large models, big batches. TurboSpec [13] runs the loop closed: a controller watches acceptance rates and system load per request and adjusts speculation length on the fly, maximizing goodput (tokens of useful work per second) rather than per-request latency. Speculation, in other words, is not just an algorithm — it is a scheduling policy.

Agents as speculation: teams, cascades, and token economics

Zoom out from the token to the task and the same division of labor appears, now wearing an org chart.

From debate to assembly lines

Multiagent debate [14] runs several instances of a model (optionally different models) that answer, then critique each other’s answers, then revise, over a few rounds. The reported effect is better factuality and reasoning than a single agent reflecting on itself — mutual verification beats self-verification, which Self-Refine [23] had shown to already help, because a critic that shares an author’s blind spot is no critic at all. Debate needs diversity and persuasion to work; conformist panels converge on confident nonsense. CAMEL [15] formalized the complementary pattern: two role-playing agents (an instructing “user” and an executing “assistant”) complete tasks autonomously, useful both as a task-solving scheme and as a machine for generating instruction data — though the authors document “task divergence,” where role-play wanders off the assignment. AutoGen [16] generalized the whole genre into “conversation programming”: agents are customizable (LLM, tool-using, human-in-the-loop), and the application is the conversation protocol among them. MetaGPT [17] imported the assembly line: standardized operating procedures, a shared message pool, and specialized roles, so that a software task moves down a line like a product — one agent’s verified output is the next agent’s input.

Notice what all four share structurally: generation is parallel and cheap per unit; verification/coordination is the funnel. The debate round, the aggregator, the message pool — these are the referee.

Heterogeneous teams and token economics

Once agents differ in strength, the loop acquires an economics. FrugalGPT [18] cascades models cheap-to-expensive: a small model answers, a scorer decides whether to escalate, and only hard queries pay for the frontier model — reporting cost cuts of up to about an order of magnitude on some benchmarks at matched quality. RouteLLM [19] learns the escalation decision from preference data, a router trained to send each query to the cheapest model likely to satisfy it, roughly tripling cost-efficiency at modest quality loss in the authors’ evaluation. These are speculative decoding’s α and c in dollar clothing: escalate too eagerly and you pay 1 + γ c for nothing; escalate too rarely and quality bleeds.

The complementary axis is width: spend more cheap proposals instead of a better one. More Agents Is All You Need [20] shows majority-vote accuracy scaling with the number of sampled agents — up to a saturation point set by task difficulty. Mixture-of-Agents [21] replaces the vote with a synthesizer: several proposer models draft, an aggregator model weaves them into one answer, layered to multiple rounds — with open-weight proposer teams topping several chat benchmarks previously held by much larger single models. Large Language Monkeys [22] pushes width to its logical end: sample a hundred solutions and coverage of the correct one keeps growing, roughly log-linearly in compute — but finding which sample is right becomes the binding constraint, and the paper’s punchline is that the verifier, not the generator, is the new bottleneck. That sentence is the thesis of this essay wearing shorts.

Don’t throw away the verdict: verification becomes training signal

Here is the turn. In inference, the verdict is consumed and discarded: accepted tokens ship, rejected tokens vanish. A parallel literature asks the obvious next question — if we bothered to verify it, why not learn from it?

Rejection-sampling fine-tuning: the verifier as data filter

The simplest reuse is data hygiene. RAFT [25] fine-tunes only on self-generated samples in the top quantile of a reward model’s scores — throw away the refusals, keep the passes. ReST [26] loops it: a Grow step samples from the current policy and scores, an Improve step trains offline on the filtered subset, and repeating the two fights the distribution drift that one-shot self-training suffers. ReST^EM [27] sharpens the filter to a plain indicator — keep only samples that pass a verifier (a program runs, an answer checks) — and frames the whole thing as expectation–maximization; on competition math and code, self-training this way surpassed training on human demonstrations, with the caveat that when the verifier is soft (a learned reward), the loop degenerates into reward hacking. SPIN [28] needs no verifier beyond the data itself: train a discriminator to tell self-generated text from human text, fine-tune the generator toward the human side, repeat — a fixed point when the two distributions match. V-STaR [29] notes that the failures are also signal: train a verifier on both correct and incorrect self-generated solutions, and the verifier improves faster than the generator alone ever could. Self-Rewarding Language Models [30] makes the judge internal: the model scores its own candidates with an LLM-as-judge prompt and applies preference optimization to its own ranking, so instruction-following and judging ability climb together across iterations.

Group-relative RL and the pass-rate band

Verbal-only loops (Reflexion [24], which stores natural-language self-critique in episodic memory, belongs here too) graduate into weight updates through RL, and the modern recipe is lean. GRPO [31] drops the learned value network entirely: for each prompt sample a group of G responses, score them, and standardize the reward within the group to get each sample’s advantage,

then apply a clipped, KL-regularized policy gradient. The group is the baseline.

Now notice a structural fact that becomes the whole story. If every response in the group earns the same reward — all pass, or all fail — then every advantage is zero and the gradient is exactly zero. The prompt taught nothing. DAPO [32] turns this bug into a budget with dynamic sampling: over-sample prompts, keep only those whose group has non-degenerate reward variance, and pad the batch until it is full. In pass-rate terms: prompts the policy always solves (pass rate 1) and prompts it never solves (pass rate 0) are discarded; gradients are released only for pass rates strictly inside (0, 1) — the band where the policy sometimes fails. This is just-in-time allocation of learning: compute is spent precisely at the current frontier of the policy’s competence, and as the policy learns, the band moves with it. Easy tasks graduate out of the curriculum automatically; currently impossible tasks wait outside until they become merely hard.

Test-time adaptation: just-in-time RL

The newest step moves the adaptation into deployment. TTRL [33] runs RL directly on unlabeled test-time traffic, using majority voting across sampled responses as a pseudo-reward — no ground truth — and reports reasoning gains beyond what the frozen model (or plain majority voting at test time) achieves. And the loop’s most explicit namesake, JitRL [34], closes the circle: a deployed agent with frozen weights maintains a non-parametric memory ℳ of experienced ⟨ s, a, r ⟩ triplets. At each new state s, it retrieves the relevant neighborhood 𝒩(s), uses the stored returns to estimate an advantage Â(s, a) for each candidate action, and adjusts the base model’s logits in closed form,

an update the paper proves is the exact closed-form solution of a KL-constrained policy-improvement objective — improvement without a single gradient step. After each episode, an evaluator computes discounted returns G_t and writes the new trajectories back into ℳ. On web-agent and text-game benchmarks this training-free loop beats expensive fine-tuning baselines at a reported 30×+ cost reduction, because “learning” is a retrieval plus a vector addition that arrives just in time — at the moment of use, for the state actually being faced, using exactly the experiences that matter to it.

Where the loop strains

The unified loop is a lens, and lenses have blind spots. Four strains are visible in the literature itself.

The verifier is the ceiling. Speculative decoding is safe because its verifier is the target model — an oracle by construction. Every downstream cousin substitutes a softer referee: a judge, a vote, a reward model, a majority. Soft referees are wrong at exactly the states where help is needed most. ReST^EM degenerates under soft rewards [27]; majorities are confidently wrong on hard or adversarial distributions; self-judges share the judged’s blind spots [30]. The stronger the adaptation, the more a wrong verdict costs — an accepted-wrong draft wastes tokens, a trained-on-wrong-sample wastes weights.

The loop drifts. Adaptation moves the policy, and everything tuned to the old policy goes stale. A drafter fine-tuned to yesterday’s target sees its acceptance rate α decay; EAGLE-3’s training-time test is a patch for exactly this train–serve mismatch; a debate panel that adapts together risks converging on shared error. Loops need re-tracking, or the α that justified them quietly evaporates.

Verification does not scale for free. Tree attention pays memory for breadth; n-agent teams pay tokens linearly while coverage grows only log-linearly [22]; goodput controllers exist because speculation itself has an optimal length that moves with load [13]. The economics keep bending toward verify less, but verify better.

Adaptation at test time is a governance problem. TTRL and JitRL update the deployed artifact from unlabeled traffic. The guardrails on display — KL constraints, closed-form bounded updates, band-labeled data release — are the field’s current honest answer, not a settled one. A system that learns at the moment of use is also a system that can be taught at the moment of use.

One loop, three timescales: a comparison

System family	Cheap proposer	Careful verifier	Where the verdict goes
Speculative decoding [1, 2]	Small or auxiliary drafter q_φ	Target model, one batched pass	Accept/reject prefix; then discarded
Self-speculation [3, 4]	Same model, early layers + early exit	Same model, full depth	Same — with zero drafter memory
Medusa / EAGLE [5–8]	Extra heads; feature-level autoregression; dynamic trees	Target with tree attention	Same — tree confidence reshapes the next draft
Retrieval / n-gram drafting [9, 10]	Datastore or Jacobi-style parallel pool	Target model	Same — no training anywhere
Debate, MoA, majority voting [14, 20, 21]	Several agents in parallel	Mutual critique; vote; aggregator	Merged into the final answer
Cascades and routers [18, 19]	Small model answers first	Scorer or learned router decides escalation	Sets the dollar budget for the next query
Rejection-sampling fine-tuning [25–30]	The policy samples broadly	Reward model, verifier, or self-judge	Filters the training set; updates weights offline
Group-relative RL [31, 32]	The policy itself, G times	Reward + within-group standardization	Gradients released only in the pass-rate band (0,1)
Test-time RL, JitRL [33, 34]	Deployed policy samples	Majority vote; retrieved experience	Logits shifted in closed form, just in time

Read down the last column and the arc of the field is plain: the verdict went from ephemeral (accept and forget) to aggregated (merge and answer) to accumulated (filter and train) to immediate (shift this very logit). The cheap draft and the careful referee are the same two characters in every scene; only the timescale of their consequence changes.

References

[1] “Fast Inference from Transformers via Speculative Decoding.” 2023. arXiv:2211.17192

[2] “Accelerating Large Language Model Decoding with Speculative Sampling.” 2023. arXiv:2302.01318

[3] “Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding.” 2023. arXiv:2309.08168

[4] “LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding.” 2024. arXiv:2404.16710

[5] “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.” 2024. arXiv:2401.10774

[6] “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.” 2024. arXiv:2401.15077

[7] “EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees.” 2024. arXiv:2406.16858

[8] “EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test.” 2025. arXiv:2503.01840

[9] “REST: Retrieval-Based Speculative Decoding.” 2023. arXiv:2311.08252

[10] “Break the Sequential Dependency of LLM Inference Using Lookahead Decoding.” 2024. arXiv:2402.02057

[11] “Contrastive Decoding: Open-ended Text Generation as Optimization.” 2022. arXiv:2210.15097

[12] “Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding.” 2024. arXiv:2402.12374

[13] “TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput.” 2024. arXiv:2406.14066

[14] “Improving Factuality and Reasoning in Language Models through Multiagent Debate.” 2023. arXiv:2305.14325

[15] “CAMEL: Communicative Agents for \”Mind\” Exploration of Large Language Model Society.” 2023. arXiv:2303.17760

[16] “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation.” 2023. arXiv:2308.08155

[17] “MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.” 2023. arXiv:2308.00352

[18] “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.” 2023. arXiv:2305.05176

[19] “RouteLLM: Learning to Route LLMs with Preference Data.” 2024. arXiv:2406.18665

[20] “More Agents Is All You Need.” 2024. arXiv:2402.05120

[21] “Mixture-of-Agents Enhances Large Language Model Capabilities.” 2024. arXiv:2406.04692

[22] “Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.” 2024. arXiv:2407.21787

[23] “Self-Refine: Iterative Refinement with Self-Feedback.” 2023. arXiv:2303.17651

[24] “Reflexion: Language Agents with Verbal Reinforcement Learning.” 2023. arXiv:2303.11366

[25] “RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment.” 2023. arXiv:2304.06767

[26] “Reinforced Self-Training (ReST) for Language Modeling.” 2023. arXiv:2308.08998

[27] “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models.” 2023. arXiv:2312.06585

[28] “Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.” 2024. arXiv:2401.01335

[29] “V-STaR: Training Verifiers for Self-Taught Reasoners.” 2024. arXiv:2402.06457

[30] “Self-Rewarding Language Models.” 2024. arXiv:2401.10020

[31] “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” 2024. arXiv:2402.03300

[32] “DAPO: An Open-Source LLM Reinforcement Learning System at Scale.” 2025. arXiv:2503.14476

[33] “TTRL: Test-Time Reinforcement Learning.” 2025. arXiv:2504.16084

[34] “Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates.” 2026. arXiv:2601.18510