SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-speculative-decoding

Speculative decoding speeds up a big model by drafting tokens with a smaller one. What happens to the output?

It stays identical, just 2 to 3 times faster in the paper's test.

Start playing free

  1. It gets slightly worse
  2. It becomes the small model's output
  3. It stays exactly the same ✔

Why: Normally K tokens need K one-after-another runs of the big model. A fast small model guesses ahead, and the big model checks several guesses at once, keeping only what it would have produced anyway.

Huh, didn't know that: On T5-XXL, the 2022 paper measured a 2X to 3X speedup with identical outputs.

The dad joke
Speculative decoding is like a kid finishing Dad's sentences. Dad just nods when they get it right.
Like this card?
Play the game

References

Answer: It stays exactly the same. Normally K tokens need K one-after-another runs of the big model. A fast small model guesses ahead, and the big model checks several guesses at once, keeping only what it would have produced anyway.

  1. Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding (arXiv 2211.17192) arxiv.org/abs/2211.17192
    show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhich latency measure matters most for how fast a streaming chatbot feels?Artificial IntelligenceIn machine learning, what is inference?Artificial IntelligenceHow is standard dropout commonly handled at inference time?

That's stuff everyone should know.