How it works
Regularization is the set of techniques that prevent a machine learning model from memorizing (overfitting) the training data. A model with overfitting learns specific patterns from training, including noise, and does not generalize to new data.
The most common techniques are: weight penalization (weight decay / L2), which discourages very large weights; dropout, which randomly turns off neurons; and early stopping, which halts training when performance on validation data stops improving. Data augmentation (increasing training data with variations) and reducing model complexity also count.
Practical example
You train a neural network to classify images of cats and dogs. Without regularization, the network can memorize the training images and fail on new photos. With weight decay and dropout, the network learns general features (ears, snout) instead of memorizing pixels, and performs better on images it has never seen.
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)What it's for
Regularization is essential for models to work outside the training environment. Without it, complex models tend to overfit, especially with little data. Choosing the technique and its intensity (hyperparameter) is a central part of training and directly impacts the final model quality.