How it works
Dropout is a regularization technique used in neural networks. During training, at each step, a fraction of the neurons in a layer is randomly turned off, or zeroed out. This prevents the network from relying too heavily on a single neuron or a specific path, forcing it to learn more robust representations.
The dropout rate, for example 0.2 or 0.5, defines the proportion of neurons that are turned off. At inference time, dropout is disabled and the weights are adjusted, or scaled, to compensate for the difference between training and use.
Practical example
Imagine a neural network that classifies emails as spam. Without dropout, the network can memorize specific patterns from the training set, overfitting, and fail on new emails. With a dropout of 0.3, 30% of the neurons are turned off at each step, and the network learns to recognize spam from several independent signals, generalizing better.
In code with PyTorch:
import torch.nn as nn
layer = nn.Dropout(p=0.3)What it's for
Dropout is used to reduce overfitting and improve the generalization of neural networks. It is one of the simplest and most effective techniques, used in vision, language and tabular models. It is often combined with other forms of regularization, such as weight decay and data augmentation.