SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-multihead-subspaces

What is a key benefit of using multiple attention heads?

Heads can attend to different representation subspaces.

Start playing free

  1. They remove every parameter
  2. They force identical scores
  3. They capture different subspaces ✔

Why: Each head has its own learned projections of queries, keys, and values. Parallel heads can therefore focus on different positions and kinds of relationships.

Huh, didn't know that: The original base Transformer used eight attention heads, each with 64-dimensional keys and values.

The dad joke
Multi-head attention sees several points at once, a real head start.
Like this card?
Play the game

References

Answer: They capture different subspaces. Each head has its own learned projections of queries, keys, and values. Parallel heads can therefore focus on different positions and kinds of relationships.

  1. Attention Is All You Need arxiv.org/html/1706.03762
    jointly attend to information from different representation subspaces at different positions
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat does a Transformer encoder produce from an input sequence?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?Artificial IntelligenceWhat did Alan Turing call his 1950 test of machine intelligence?

That's stuff everyone should know.