Hyperparameter

A hyperparameter is a setting you choose before training. It controls learning. It is not a weight the optimizer learns from data. Weights and biases move during training. Learning rate, batch size, depth, and dropout stay put unless you change them.

Learning rate is the one that wrecks runs. Too high and training diverges. Too low and it stalls. Batch size changes gradient noise and memory use. Layer count and hidden size set capacity. Dropout sets how hard you regularize. The optimizer changes how you converge. Getting these right is often as important as the architecture.

Search is expensive. Grid search tries every combo from a list and dies as the list grows. Random search samples from distributions and often finds a good point faster for the same compute. Bayesian optimization models how settings map to scores and picks the next trial. Population-based training changes schedules while the run is going.

Transfer means copying values that worked on a similar problem.

In practice people use schedulers, warmup that raises the rate at the start, and tuning services. Bad hyperparameters burn GPUs on failed runs. The model never gets a chance to be good.

Log every run. A good architecture with a bad rate looks like a bad architecture. Most failed trainings are settings, not ideas. Learning rate, batch size, and depth are hyperparameters. You pick them. The model does not learn them as weights.