CLIP (Contrastive Language-Image Pre-training) is a multimodal model from OpenAI that learns to connect images and text in a shared embedding space through contrastive learning on 400 million image-text pairs scraped from the internet.
The architecture consists of separate image and text encoders (typically a Vision Transformer and a text transformer) trained to produce aligned embeddings: an image and its caption should map to nearby points, while unrelated image-text pairs should map far apart. This enables powerful zero-shot capabilities.
To classify an image, encode the image and encode text descriptions of each class ('a photo of a dog', 'a photo of a cat'), then choose the class whose text embedding is closest to the image embedding. No task-specific training required. CLIP achieves competitive accuracy on many benchmarks without seeing any labeled examples from those datasets.
Beyond classification, CLIP enables semantic image search (find images matching a text query), image generation guidance (DALL-E uses CLIP to guide generation toward prompts), and multimodal understanding. Natural language provides richer supervision than discrete labels: 'a golden retriever playing fetch in a sunny park' contains more information than the label 'dog'.
CLIP's web-scale training captures this richness. Limitations include biases from web data and struggles with fine-grained distinctions.
Interactive Visualizer
CLIP: Contrastive Language-Image Pre-training
Interactive visualization of how CLIP learns to align images and text in a shared embedding space
Image Encoder (Vision Transformer)
Shared Embedding Space
Text Encoder (Transformer)
Contrastive Learning Process
CLIP learns by maximizing similarity between correct image-text pairs (green, positive examples) while minimizing similarity between incorrect pairs (red, negative examples).