Contrastive Learning

Contrastive learning pulls similar items together and pushes dissimilar items apart in embedding space, without labels.

Take an anchor. Define positives that should sit nearby and negatives that should sit far away. Train so anchor-positive distance is small and anchor-negative distance is large. For images, positives are often augmented views of the same photo, cropped, rotated, or color-shifted, and negatives are other photos. Features that survive those augmentations tend to be semantically useful.

SimCLR and MoCo showed this for computer vision: the embeddings transfer to downstream tasks without labeled pretraining.

CLIP treats image-caption pairs as positives and unpaired items as negatives, aligning image and text spaces. The usual loss is InfoNCE: maximize the log-probability of picking the true positive among many negatives. More negatives, especially hard negatives that are similar but not the same, usually help.

Contrastive learning is now a standard way to learn representations for vision and multimodal models.

InfoNCE looks like a softmax over one positive and many negatives in embedding space. SimCLR builds positives with augmentations in the same batch. MoCo keeps a queue of negatives so the negative set can be large. CLIP’s positive is a matched image-caption pair; a batch of N pairs yields N positives and N(N-1) negatives.

Hard negatives, such as two similar dog breeds, force the embedding to keep fine differences. That is why negative count and negative difficulty both change the representation. SimCLR (2020) pulls two augmented views of the same image together and pushes other images apart. CLIP uses the same idea across image and text.