---
title: CLIP
description: Page on Vedang Vatsa's site: https://veda.ng/glossary/clip
canonical: https://veda.ng/glossary/clip
last_updated: 2026-10-03
type: text/markdown
---
# CLIP

Source: https://veda.ng/glossary/clip
Author: Vedang Vatsa (https://veda.ng/about)

CLIP is an OpenAI model that maps images and text into one embedding space using contrastive learning.

An image encoder, typically a Vision Transformer, and a text transformer are trained so an image and its caption land nearby, while unrelated pairs land far apart. Zero-shot classification follows: encode the image, encode class phrases such as "a photo of a dog" and "a photo of a cat," and pick the closest text embedding. No labels from the target dataset are required. CLIP is competitive on many benchmarks without seeing those labeled sets.

The same embeddings support semantic image search, generation guidance (DALL-E uses CLIP to steer toward prompts), and multimodal understanding. Natural language supervision is richer than a discrete label. "A golden retriever playing fetch in a sunny park" carries more than the class "dog." Web-scale training captures that detail. Limits include biases in web data and weak performance on fine-grained distinctions.

At training, a batch of image-text pairs is encoded. Cosine similarities form an N by N matrix. The loss raises the diagonal, matched pairs, and lowers the off-diagonal. At zero-shot test time you write one phrase per class, encode those phrases once, encode the image, and argmax cosine similarity. DALL-E uses CLIP scores as a guide toward the prompt. Search uses the same cosine as a ranking score. Fine-grained classes such as bird species still trip it, and web scrapes bake in social biases. OpenAI's CLIP (2021) trained image and text encoders together on 400 million pairs so captions and pictures share a vector space.

Glossary index: https://veda.ng/glossary