Image Segmentation

Image segmentation assigns a class label to every pixel, giving dense, pixel-level understanding instead of one label or a box.

Semantic segmentation labels classes such as road, car, and person without separating instances. Instance segmentation gives each object its own mask, car 1 versus car 2 versus car 3. Panoptic segmentation does both: class labels for stuff such as sky and road, and instance masks for things such as individual cars and people.

U-Net, an encoder-decoder with skip connections, became the default for medical image segmentation. Mask R-CNN adds a mask branch to Faster R-CNN for instance segmentation. Segment Anything Model (SAM) learns general segmentation that transfers across domains.

Uses include tumor boundaries and organ outlines in medical imaging, road-scene parsing for driving, land-use maps from satellite photos, background removal and object selection in editors, and live scene parsing in augmented reality. mean Intersection over Union (mIoU) compares predicted masks to ground truth.

Pixel labels are expensive, so weakly supervised and self-supervised segmentation are active research.

Every pixel gets a label, so output tensors match image height and width. U-Net skip connections copy encoder detail into the decoder so tumor edges stay sharp. Mask R-CNN predicts a small mask per detected box. SAM, the Segment Anything Model, is the transfer-heavy extreme: prompt a mask instead of training a new head per dataset. mIoU penalizes both missed pixels and extra pixels.

Because tracing every tumor boundary by hand is slow, weakly supervised methods try to learn from cheaper labels such as boxes or scribbles. U-Net (2015) is still the default for medical pixel labels. SAM (2023) made promptable masks mainstream.