Model Distillation

Train a small model to imitate a large one. Often better than training the small model from scratch, and the reason cheap models got good.

You have a large model that works and is too expensive to serve. You want a small model that does the same job.

Training the small model from scratch on the original data gives mediocre results. Training it to imitate the large model gives much better ones.

That is distillation, and it explains a lot about why small models improved so quickly.

Why imitation beats the original data

The insight is about what the large model’s output contains.

Original training data gives one correct answer per example. The large model gives a full probability distribution over every possible answer — and that distribution encodes far more information.

Shown an image of a wolf, a label says “wolf.” A trained model says: wolf 0.7, dog 0.2, coyote 0.08, cat 0.001. That tells the student not just the answer but the structure of the problem — which mistakes are reasonable, which are absurd, how confident to be.

These are sometimes called soft targets, and learning from them is a richer signal than learning from labels. The student gets the teacher’s uncertainty, not just its conclusion.

Three approaches

Logit distillation trains the student to match the teacher’s full output distribution. Requires access to the teacher’s raw scores, so it needs open weights.

Sequence-level distillation has the teacher generate outputs, then trains the student on those as ordinary fine-tuning targets. Only needs the teacher’s text — which is why this is the practical method for closed models, and by far the most common in practice.

Reasoning distillation is the recent one that matters. Have a reasoning model produce full step-by-step traces, then train a smaller model on those traces. The student learns to reason at length, not just to answer. This produced a real jump in small-model capability on math and code, because the traces contain the intermediate structure that a bare answer omits.

Where the gains are large

Narrow tasks. A small model distilled for one classification job can approach a much larger prompted model at a fraction of the cost per call.

Latency. Small models generate faster, which changes what is possible in interactive interfaces.

On-device deployment. Where the large model simply cannot run.

Cost at volume. The whole argument, at scale.

Limits

The teacher is the ceiling. A student rarely exceeds its teacher on the distilled distribution. Errors get inherited, including hallucination patterns.

Breadth does not survive compression. A small student can match a large teacher on a narrow task and will not match it across everything. Distilling general capability into a much smaller model loses the long tail — the unusual cases the large model handled.

Distribution matters. The student learns what the teacher was asked. Prompts outside that distribution at serving time get much worse results, so the distillation set has to reflect real traffic.

Terms of service. Using a hosted model’s outputs to train a competing model is commonly prohibited. Worth reading before building on it.

Practical shape

The usual recipe is unglamorous and works: collect real production inputs, run the large model on them, keep the outputs, fine-tune a small model on those pairs, and evaluate against the teacher on held-out cases.

Two things make the difference. Use real inputs, since synthetic ones are cleaner than reality and the student inherits that gap. And filter the teacher’s outputs — verify what can be verified, drop what fails. A teacher’s mistakes become the student’s training data otherwise.

For reasoning distillation specifically, keeping only traces that reached verifiably correct answers matters a great deal. This is where synthetic data practice and distillation overlap.

What to remember

• Distillation trains a small student to imitate a large teacher, which beats training on original labels.
• The teacher’s full distribution carries problem structure — which errors are plausible — that a single label lacks.
• Sequence-level distillation (train on teacher outputs) is the practical method; reasoning distillation on step-by-step traces drove recent small-model gains.
• The teacher is a ceiling, breadth is lost, and the distillation distribution must match real traffic.
• Filter teacher outputs by verification, or you train on its mistakes.

Next: Synthetic Data