SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-attention-scale-root-dk

By what quantity does scaled dot-product attention divide query-key dot products?

It divides by the square root of key dimension.

Start playing free

  1. The sequence length squared
  2. sqrt(d_k) ✔
  3. The number of layers

Why: Dot-product magnitudes tend to grow with key dimension. Dividing by sqrt(d_k) keeps softmax inputs in a range with more useful gradients.

Huh, didn't know that: Without scaling, large dot products can push softmax toward regions with extremely small gradients.

The dad joke
Attention took the square root to keep its scores grounded.
Like this card?
Play the game

References

Answer: sqrt(d_k). Dot-product magnitudes tend to grow with key dimension. Dividing by sqrt(d_k) keeps softmax inputs in a range with more useful gradients.

  1. Attention Is All You Need arxiv.org/html/1706.03762
    divide each by sqrt(d_k), and apply a softmax function
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhich 2017 paper introduced the Transformer architecture?Artificial IntelligenceWhich two sequence mechanisms did the original Transformer dispense with?Artificial IntelligenceAfter query-key scores become weights, what does attention combine?

That's stuff everyone should know.