Hyperparameter Tuning

Some settings are learned during training; others you must choose before it starts. How to search the ones you choose without fooling yourself or wasting compute.

Training learns some numbers for you — the weights. Others you have to pick before training starts: the learning rate, the amount of regularization, the batch size, the number of layers. These are hyperparameters, and choosing them well is a search problem with two traps: wasting compute, and fooling yourself into thinking you did better than you did.

Parameters versus hyperparameters

The distinction is clean and worth stating precisely. Parameters are learned by the training process from data. Hyperparameters configure the training process itself and are set from outside it.

Learning rate, regularization strength, batch size, number of epochs, model depth and width — none of these are learned by gradient descent. They shape how learning happens. You cannot optimize them the way you optimize weights, because they sit above the loss the weights minimize. So you search.

The three axes of a search

Every tuning method is answering three questions. Naming them makes the whole activity legible.

What to search. Not all hyperparameters matter equally. Learning rate almost always dominates; many others barely move the result. Spending your budget on the influential few beats spreading it thin. If you tune one thing, tune the learning rate.

Where to look. Ranges matter, and many hyperparameters are best searched on a log scale — try 0.1, 0.01, 0.001, not 0.1, 0.2, 0.3. A learning rate’s effect is multiplicative, so even spacing in raw units clusters all your samples in the wrong region.

How to sample the space. This is where the named methods live.

Search methods

Grid search. Try every combination on a predefined grid. Simple and exhaustive, but the cost explodes: three values across four hyperparameters is 81 runs, and it wastes effort testing many values of parameters that do not matter.

Random search. Sample combinations at random within your ranges. Counterintuitively, this usually beats grid search for the same budget. The reason is sharp: if only one hyperparameter really matters, grid search tests it at only a few distinct values (the rest of each run varies the parameters that do not matter), while random search tests it at a different value every single run. You get far more resolution on the axis that counts.

Guided search. Methods that use past results to decide what to try next, concentrating samples where good results appeared instead of sampling blindly. More sample-efficient, more machinery to run. Worth it when each run is expensive.

For most work, random search over the two or three hyperparameters that matter, on log scales, gets you most of the benefit with the least ceremony.

The trap: tuning on the test set

This is the mistake that quietly invalidates results, and it deserves its own section.

You need three splits, not two — the reason train/validation/test exist as separate things. Here is why tuning makes the third one non-negotiable:

• The training set fits the weights.
• The validation set picks the hyperparameters — you compare configurations on it.
• The test set estimates final performance, and it must be touched once, at the end.

The danger is subtle. Every time you choose a hyperparameter by its validation score, you leak a little information about the validation set into your model choice. Run enough configurations and you start overfitting the validation set — picking the config that got lucky on those specific examples, not the one that generalizes. Your validation score becomes optimistic.

The test set, held out and untouched through all of this, is your defense. If you tune against it, you have no honest estimate left, and you will ship something worse than your numbers claim. The rule is absolute: the test set is not a tuning signal.

Knowing when to stop

Tuning has diminishing returns, and recognizing the plateau saves enormous effort.

Most gains come early. The first few sensible choices — a reasonable learning rate, roughly right regularization — capture the bulk of the improvement. Chasing the last fraction of a percent through hundreds more runs is usually a poor trade against better data or a better model.

A bigger lever often sits elsewhere. When tuning stalls, the ceiling is frequently the data or the model architecture, not the hyperparameters. More or cleaner data reliably beats a heroic search over a fixed dataset.

Budget the search up front. Decide how many runs you will spend before starting, so the search ends by plan rather than by exhaustion. Open-ended tuning expands to fill whatever time it is given.

What to remember

• Parameters are learned from data; hyperparameters configure training and must be chosen by search.
• Spend your budget on the few hyperparameters that matter — the learning rate above all — and search them on a log scale.
• Random search beats grid search at equal budget because it tests the influential axis at more distinct values.
• Keep three splits: train fits weights, validation picks hyperparameters, and the test set is touched once — tuning on it destroys your honest estimate.
• Gains come early; when tuning plateaus, better data or a better model usually beats more search.

Next: Why Evaluation Is Hard