Hyperparameter Tuning

Hyperparameter tuning is choosing the settings that control how a machine-learning model learns, before weights are trained.

Weights are learned during training. Hyperparameters are set first: learning rate, tree depth, number of hidden units, regularization strength. The goal is the combination that scores best on unseen data. Grid search tries all listed combinations. Random search samples combinations. Bayesian optimization uses past trials to pick the next trial.

Tuning can move a weak model into a usable one. Many algorithms are sensitive to these knobs. A well-tuned random forest might reach 90% accuracy on a diagnosis task where defaults sit at 70%. In finance, healthcare, and driving, those points map to cost, patient outcomes, and safety.

Learning rate, depth, hidden size, and regularization are not learned by gradient descent on the main loss; you choose them, train, and measure held-out performance. Grid search is exhaustive and expensive. Random search samples the space. Bayesian optimization treats past runs as data and proposes the next setting.

The 70% versus 90% gap on a diagnosis forest is a tuning gap, not a new algorithm. Finance, health, and driving care because those points are money, outcomes, and accidents. Tools such as Optuna and Vizier search hyperparameter space so you are not guessing the learning rate by hand.