How it works
In machine learning, data is split into three parts: training, validation and test. The training set teaches the model. The validation set helps you make decisions: which architecture to use, which hyperparameters to tune, when to stop training.
Validation works like a dress rehearsal. You train several versions of the model and compare performance on the validation set to pick the best one. After choosing, you evaluate the final model on the test set, which has never been used before.
The golden rule is: the test set is used only once, at the end. If you look at the test set multiple times to adjust the model, it becomes validation and loses its role of measuring generalization.
Practical example
Imagine training a model to classify emails as spam. You have 10 thousand labeled emails. You use 7 thousand for training, 1.5 thousand for validation and 1.5 thousand for test.
During training, you test different learning rate values and number of layers, comparing accuracy on validation. You choose the configuration with the best result on validation. At the end, you measure final accuracy on the test set, which was never touched.
If you use the test set to choose the configuration, the model can adapt to the test set and look better than it really is. This is called overfitting to the test set.
What it is for
The validation set is used to tune hyperparameters, choose between models and decide when to stop training, early stopping. Without it, you have no way to know if the changes you made are improving the model or just memorizing the training data.
Difference between validation and test
Validation is used repeatedly during development to make decisions. Test is used once, at the end, to measure final performance. Validation guides choices; test measures results.