Overfitting

Overfitting is a fundamental machine learning failure mode where a model memorizes training examples rather than learning generalizable patterns, performing excellently on training data but poorly on new data. The model learns the noise, quirks, and specific details of training examples that don't apply to the broader distribution. A model that achieves 99% training accuracy but 60% test accuracy is severely overfit, it has essentially memorized answers rather than understanding the underlying patterns. Overfitting occurs when model capacity exceeds what the data requires, when training runs too long without regularization, or when training data is too small or unrepresentative. Signs include a growing gap between training and validation loss during training, and model predictions that are overly confident. Prevention strategies attack different causes: more training data reduces the need to memorize, data augmentation artificially expands the training distribution, regularization techniques like dropout and weight decay penalize complexity, early stopping halts training before the model memorizes, and cross-validation helps detect overfitting during development. The bias-variance tradeoff frames overfitting as high variance, the model's predictions vary too much based on which specific training examples it saw. Finding the right model complexity that fits the signal without fitting the noise is the central challenge of machine learning.