How it works
A hyperparameter is a setting you choose to guide how a model is trained or run, instead of being learned directly as a weight during training. Common examples include learning rate, batch size, number of epochs, regularization strength, and number of layers in an architecture.
Model parameters and hyperparameters play different roles. Parameters, such as weights, are adjusted from the data by the training process. Hyperparameters control aspects of that process, such as the size of each update or the amount of training. Some systems also have generation hyperparameters, such as temperature, which affect how a response is produced.
Choosing a hyperparameter can change model behavior. A high learning rate can make tuning difficult; a very low one can make training slow or prevent it from making adequate progress within the available budget. A larger batch changes how examples are grouped. The effect depends on the dataset, the algorithm, and the infrastructure.
Teams usually test combinations on a validation set and compare metrics defined for the task. It is important to keep the test set separate, used for final evaluation, so you do not repeatedly choose settings based on responses from that set. Recording the settings and results makes experiments reproducible.
Practical example
Imagine you train a classifier to identify support messages by category. First, define a metric that represents the goal, such as accuracy on critical categories. Then test some learning rates and batch sizes on training data, track the metric on validation data, and record each configuration.
If a configuration performs well on training and worse on validation, the model may be memorizing examples instead of generalizing. In that case, adjust the experiment, review the data, or consider regularization techniques. Change few variables at a time when you need to understand which choice influenced the result.
Do not pick only the configuration that produced the best number in a single run. Random variations in the data or initialization can affect the measurement. Compare results consistently, check the computational cost, and keep a trail that allows you to repeat the experiment.
What it's for
Hyperparameters are used to control training, architecture, or model generation. They help adapt an approach to a problem, but they are not universal buttons: a value that works for one dataset can be unsuitable for another.
The concept is useful when tuning your own models, reproducing experiments, comparing configurations, and debugging unstable training. It also helps separate what the model learned from the choices made by whoever built the process.
To choose values, start with a reference configuration, define a metric, and run organized experiments. Record data, code, seeds when applicable, and relevant versions. The training documentation for PyTorch presents an example of hyperparameter search and evaluation on validation.