How it works
In model distillation, a large, already trained model (the teacher) generates responses or probabilities for a set of examples. A smaller model (the student) is then trained to reproduce those outputs, instead of learning directly from the original data.
The key is to use the teacher's probabilities, not just the final answer. Instead of saying "the answer is dog", the teacher says "90% dog, 5% cat, 5% wolf". That extra information, called soft labels, gives the student more signal and speeds up learning.
The process has three parts: the teacher (large, slow, expensive), the student (small, fast, cheap) and the training data. The student learns to imitate the teacher, not absolute truth. So the student's quality depends on the teacher's quality.
Practical example
Imagine you have a 70 billion parameter model that answers well, but is slow and expensive to run. With distillation, you train an 8 billion parameter model to imitate the large one's responses. The small model gets faster and uses a fraction of the memory, with a small loss in quality.
This is how "mini" variants from various providers are created. The distilled model inherits the teacher's style and knowledge, but runs on more modest hardware.
What it's for
Distillation is used to reduce cost and latency without losing much quality. It's useful when you need to run AI on device (on-device), at the edge, or on a tight budget.
It's also used to create specialized models: a large generalist model teaches a small model focused on code or customer support, for example.
Difference between distillation and fine-tuning
In fine-tuning, the model learns directly from human labeled examples. In distillation, it learns from the outputs of another model. Fine-tuning requires high quality data and is more expensive. Distillation leverages the knowledge already condensed in the teacher and is usually cheaper.