Vision Transformer

Vision Transformer, ViT, splits an image into patches, treats each patch as a token, and runs a transformer on them.

An image is cut into non-overlapping patches, typically 16x16 pixels. Each patch is flattened, given a learnable positional embedding, and passed through transformer encoder layers. A learnable classification token pools patch information for the final prediction. ViT dropped the assumption that vision needs strong image-specific inductive biases such as translation equivariance and locality.

Attention has no built-in spatial locality: every patch can attend to every other patch. With enough data, ViT matches CNNs. With very large pretraining sets, it can exceed them.

The original ViT paper showed scale is the switch: smaller ViTs lag CNNs; larger ViTs trained on more data win. That echoes language-model scaling. Shared transformer blocks then made unified vision-language models such as CLIP and GPT-4V practical. Variants include DeiT for more efficient training, Swin Transformer for hierarchical features, and BEiT for masked image modeling.

16x16 patches on a 224 image yield 196 tokens plus the class token. Positional embeddings tell the encoder where each patch sat. Self-attention then mixes any pair of patches, so a wheel at the bottom can attend to a car body at the top. DeiT improved training recipes so ViT could work with less data. Swin restores hierarchy with shifted windows.

BEiT masks patches, like masked language modeling for images. CLIP and GPT-4V reuse this patch-as-token idea so vision and language share one attention stack. Google's ViT paper splits an image into patches and runs a Transformer on them. At scale it matched CNNs.