SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-clip-image-text-pairs

What was OpenAI's CLIP trained to predict on 400 million internet images?

Which caption goes with which image.

Start playing free

  1. Which caption goes with which image ✔
  2. The next pixel in each image
  3. The camera that took each image

Why: Instead of fixed labels like dog or cat, CLIP learned from image and caption pairs. Because it links pictures to free text, you can classify images just by writing the category names.

Huh, didn't know that: CLIP matched the original ResNet-50's ImageNet accuracy zero-shot, without using any of its 1.28 million training examples.

The dad joke
CLIP spent years matching captions to pictures. Basically it's every grandparent on social media.
Like this card?
Play the game

References

Answer: Which caption goes with which image. Instead of fixed labels like dog or cat, CLIP learned from image and caption pairs. Because it links pictures to free text, you can classify images just by writing the category names.

  1. Radford et al., Learning Transferable Visual Models From Natural Language Supervision (arXiv 2103.00020) arxiv.org/abs/2103.00020
    the simple pre-training task of predicting which caption goes with which image
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceHow does a Vision Transformer (ViT) feed an image into a Transformer?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?Artificial IntelligenceWhat did Alan Turing call his 1950 test of machine intelligence?

That's stuff everyone should know.