SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-decoder-causal-mask

Why does an autoregressive Transformer decoder mask later positions?

It prevents a token from seeing future outputs.

Start playing free

  1. To block future information ✔
  2. To remove all attention
  3. To increase vocabulary size

Why: During next-token prediction, position i should depend only on earlier known outputs. A causal mask blocks attention links to subsequent positions.

Huh, didn't know that: The paper implements illegal attention links by setting their pre-softmax values to negative infinity.

The dad joke
The decoder wore a mask to avoid spoilers.
Like this card?
Play the game

References

Answer: To block future information. During next-token prediction, position i should depend only on earlier known outputs. A causal mask blocks attention links to subsequent positions.

  1. Attention Is All You Need arxiv.org/html/1706.03762
    prevent positions from attending to subsequent positions
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceAt its core, what is a large language model trained to do?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?Artificial IntelligenceWhat did Alan Turing call his 1950 test of machine intelligence?

That's stuff everyone should know.