Overfitting

Overfitting is when a model memorizes the training set instead of learning patterns that hold on new data. Training scores look great. Fresh examples look bad.

The model has learned noise and quirks that only exist in those particular rows. A model at 99% training accuracy and 60% test accuracy is badly overfit. It has stored answers. It has not learned the structure.

It happens when the model is too large for the data, when you train too long with no regularizer, or when the training set is small or skewed. Watch the gap between training loss and validation loss. Watch overconfident predictions on the train set.

Fixes attack the cause. More data reduces the need to memorize. Augmentation widens the training distribution. Dropout and weight decay penalize complexity. Early stopping ends the run before memorization. Cross-validation catches the gap during development. In bias-variance language, overfitting is high variance: predictions swing too hard based on which examples you happened to train on.

The job in machine learning is to fit the signal and leave the noise.

A held-out test set is the check. If you tune on the test set, you have overfit that too. Keep a final split you do not touch until the end. Data leakage is overfitting with extra steps: the model saw the answer during training and you called it skill. Overfitting means train loss is low and test loss is high. Dropout, weight decay, and more data are the usual fixes.