SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-sinusoidal-position-frequencies

How did the original Transformer construct its fixed positional encodings?

It used sines and cosines of different frequencies.

Start playing free

  1. Random class labels
  2. Sines and cosines ✔
  3. One-hot output classes

Why: Different embedding dimensions receive sinusoidal signals with different wavelengths. Together, those values give each sequence position a structured code.

Huh, didn't know that: Even dimensions use sine and odd dimensions use cosine in the paper's formula.

The dad joke
Positional encoding made waves in every dimension.
Like this card?
Play the game

References

Answer: Sines and cosines. Different embedding dimensions receive sinusoidal signals with different wavelengths. Together, those values give each sequence position a structured code.

  1. Attention Is All You Need arxiv.org/html/1706.03762
    we use sine and cosine functions of different frequencies
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhy does a Transformer add positional information to token embeddings?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?Artificial IntelligenceWhat did Alan Turing call his 1950 test of machine intelligence?

That's stuff everyone should know.