CLIP is an OpenAI model that maps images and text into one embedding space using contrastive learning.
An image encoder, typically a Vision Transformer, and a text transformer are trained so an image and its caption land nearby, while unrelated pairs land far apart. Zero-shot classification follows: encode the image, encode class phrases such as "a photo of a dog" and "a photo of a cat," and pick the closest text embedding. No labels from the target dataset are required.
CLIP is competitive on many benchmarks without seeing those labeled sets.
The same embeddings support semantic image search, generation guidance (DALL-E uses CLIP to steer toward prompts), and multimodal understanding. Natural language supervision is richer than a discrete label. " Web-scale training captures that detail. Limits include biases in web data and weak performance on fine-grained distinctions.
At training, a batch of image-text pairs is encoded. Cosine similarities form an N by N matrix. The loss raises the diagonal, matched pairs, and lowers the off-diagonal. At zero-shot test time you write one phrase per class, encode those phrases once, encode the image, and argmax cosine similarity. DALL-E uses CLIP scores as a guide toward the prompt.
Search uses the same cosine as a ranking score. Fine-grained classes such as bird species still trip it, and web scrapes bake in social biases. OpenAI's CLIP (2021) trained image and text encoders together on 400 million pairs so captions and pictures share a vector space.
CLIP: Contrastive Language-Image Pre-training
Interactive visualization of how CLIP learns to align images and text in a shared embedding space
Image Encoder (Vision Transformer)
Shared Embedding Space
Text Encoder (Transformer)
Contrastive Learning Process
CLIP learns by maximizing similarity between correct image-text pairs (green, positive examples) while minimizing similarity between incorrect pairs (red, negative examples).