Gradient Descent

Gradient descent trains a neural network by moving the weights a little, again and again, in the direction that lowers the loss. The gradient of the loss points toward higher error. You step the other way by a learning rate. Repeat until the loss stops falling.

Batch gradient descent uses the whole training set for each step. Accurate, slow on large data. Stochastic gradient descent uses one random example. Fast, noisy. Mini-batch gradient descent, the usual default, uses a small batch. That balances noise and speed and maps well onto GPUs.

The learning rate matters. Too large and you overshoot, oscillate, or blow up. Too small and training crawls or sticks. Schedules start with larger steps and shrink later. Adam, AdaGrad, and RMSprop keep a learning rate per parameter and adjust it from gradient history. Momentum adds inertia so noisy steps do not jerk the run around, and so the optimizer can roll past shallow dips.

Modern deep learning is these variants, run over billions of weights. The loss surface is not a simple bowl. The algorithm still works because small steps, good batching, and adaptive rates get you to a useful minimum, not because the math is pretty.

Second-order methods exist and are rarely used at this scale. The industry bet is first-order steps plus good defaults, run for a long time. Gradient descent walks downhill on the loss. SGD, Adam, and AdamW are the usual variants in deep learning.