SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-cross-attention-sources

In Transformer cross-attention, where do queries and key-value pairs come from?

Queries come from the decoder, keys and values from the encoder.

Start playing free

  1. All come from the tokenizer
  2. Decoder queries, encoder keys and values ✔
  3. Encoder queries, dataset keys only

Why: The decoder asks what input information is relevant using its queries. The encoder output supplies the keys to match and values to retrieve.

Huh, didn't know that: This lets every decoder position attend over all encoded input positions.

The dad joke
The decoder asked the questions, and the encoder had the key values.
Like this card?
Play the game

References

Answer: Decoder queries, encoder keys and values. The decoder asks what input information is relevant using its queries. The encoder output supplies the keys to match and values to retrieve.

  1. Attention Is All You Need arxiv.org/html/1706.03762
    queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat makes an attention mechanism self-attention?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?Artificial IntelligenceWhat did Alan Turing call his 1950 test of machine intelligence?

That's stuff everyone should know.