SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-position-needed-without-recurrence

Why does a Transformer add positional information to token embeddings?

Attention alone does not encode sequence order.

Start playing free

  1. To represent token order ✔
  2. To eliminate embeddings
  3. To count model parameters

Why: The original architecture has no recurrence or convolution to carry position implicitly. Positional encodings tell it where tokens occur in the sequence.

Huh, didn't know that: The positional and token embeddings share a dimension so they can be added.

The dad joke
The tokens needed positions because nobody liked unassigned seating.
Like this card?
Play the game

References

Answer: To represent token order. The original architecture has no recurrence or convolution to carry position implicitly. Positional encodings tell it where tokens occur in the sequence.

  1. Attention Is All You Need arxiv.org/html/1706.03762
    we must inject some information about the relative or absolute position of the tokens
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat does a learned embedding assign to each input token?Artificial IntelligenceWhat property does cosine similarity ignore when comparing two embeddings?Artificial IntelligenceHow did the original Transformer construct its fixed positional encodings?

That's stuff everyone should know.