SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-self-attention-same-sequence

What makes an attention mechanism self-attention?

It relates positions within the same sequence.

Start playing free

  1. It changes its optimizer
  2. It labels its own data
  3. Queries, keys, and values share a sequence ✔

Why: In self-attention, the sequence supplies the queries, keys, and values. Each position can build a representation using information from other allowed positions.

Huh, didn't know that: Encoder self-attention can connect every position to every position in the previous encoder layer.

The dad joke
Self-attention looked inward and found the whole sequence.
Like this card?
Play the game

References

Answer: Queries, keys, and values share a sequence. In self-attention, the sequence supplies the queries, keys, and values. Each position can build a representation using information from other allowed positions.

  1. Attention Is All You Need arxiv.org/html/1706.03762
    Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat lets a recurrent neural network carry information through a sequence?Artificial IntelligenceWhat does a Transformer encoder produce from an input sequence?Artificial IntelligenceIn Transformer cross-attention, where do queries and key-value pairs come from?

That's stuff everyone should know.