Optical Character Recognition

OCR converts images of text, scans, signs, and handwriting into machine-readable characters.

Older systems isolated characters with image processing and matched them to known patterns. Modern OCR uses deep learning. CNNs extract visual features. LSTMs or transformers decode character sequences. That handles varied fonts, damaged scans, and messy layouts better than rule systems. Document OCR must parse multi-column pages, tables, headers, footnotes, and mixed content.

Scene text OCR reads street signs, product labels, and storefronts, with extra trouble from rotation, perspective, occlusion, and decorative type. Handwriting varies by person, so it is harder still.

The pipeline is usually text detection (where the text is), text recognition (what it says), and post-processing (language modeling to fix errors). TrOCR uses a vision transformer plus a text transformer and treats OCR as image-to-text translation. Uses include document digitization, automated data entry, accessibility, receipt processing, license plate recognition, and live translation overlays.

Detection boxes or polygons find text regions. Recognition then reads each region as a character sequence. Language-model post-processing fixes confusions such as 0 versus O. Scene text adds perspective and fancy fonts that scanned contracts do not have. Handwriting adds writer-specific shapes.

TrOCR’s vision transformer encodes the crop; its text transformer decodes tokens, like translation from pixels to words. Digitizing archives, receipts, plates, and AR translation all share that three-stage pipeline even when the backbone is a transformer. Tesseract is the long-running open OCR engine. Modern cloud OCR adds layout and handwriting on top of that job: pixels to text.