What is model distillation?

How it works

Classical distillation starts with a teacher model and a student model. The teacher is usually larger, more capable, or both. The student is smaller, cheaper, or easier to deploy. The idea is not to copy parameters directly. It is to train the student on the teacher's behaviour.

In the earliest forms of distillation, the teacher provided soft targets. Instead of learning only from a hard label such as "class A", the student learned from the teacher's fuller probability distribution, which contains richer information about what alternatives the teacher considered plausible. That helps the student learn class relationships and decision boundaries more effectively.

In language models, distillation can happen at different levels. One approach uses output text. You send representative prompts to the teacher, gather the responses, and train the student on those input-output pairs. Another approach uses token-level probabilities or intermediate representations, which can transfer more nuance if the training stack supports it. A third approach uses rationales or worked intermediate steps, so the student does not only learn the final answer but also some of the reasoning pattern behind it.

A practical distillation project usually has several stages.

First, you choose the task. Distillation works best when the target behaviour is clear. "Be generally brilliant" is too broad. "Classify incoming procurement emails into five categories and extract supplier ID if present" is much better.

Second, you define the teacher. This could be one frontier model, a strong open model, or even an ensemble of several systems. In some cases the teacher is not a single model but a combination of model output and human correction.

Third, you build the distillation dataset. This is a crucial step. The prompts used for distillation should reflect the real traffic and edge cases you care about. If the dataset is narrow, the student will be narrow. If the teacher is prompted poorly, the student may inherit that weakness. If the data is unrepresentative, the project can look good offline and disappoint in practice.

Fourth, you train the student. This may look similar to supervised fine-tuning, but the supervision comes partly or largely from the teacher. In some pipelines, distillation is integrated with fine-tuning so the student learns from both teacher outputs and ground-truth labels. In others, the teacher produces rationales as well as answers, which gives the student richer supervision.

This rationale-based approach is important because it shows that distillation is evolving. It is no longer only about compressing a model's final outputs. Some newer methods aim to transfer a more useful internal pattern of problem solving. A strong example is "distilling step-by-step", where a larger model first produces intermediate rationales and a smaller model is trained on those rationales alongside the final task labels. That can reduce the amount of labelled data needed and sometimes let a smaller model beat standard fine-tuning.

There are also architectural variants. Some distillation projects compress a model during pretraining. Others happen after pretraining for a specific task. Some methods distil intermediate layers, attention patterns, or embeddings. Others use self-distillation, where one model teaches a smaller version of itself or even a later version of itself.

Importantly, distillation does not produce magic. A student model has a capacity ceiling. It will not perfectly retain every capability of a much larger teacher, especially if the target job is broad or reasoning-intensive. Distillation is about selective transfer under constraints.

It also interacts with other efficiency methods. A team might distil first, then quantise the student for deployment. Or it might prune a model and use distillation to recover some lost quality. This is one reason distillation and small language models are related but distinct. Distillation is a method. The resulting student may become an SLM, but the method can also produce specialised mid-sized models.

In production, the project should end with evaluation, not with training. You need a gold standard test set, side-by-side comparisons with the teacher, checks for safety regressions, and realistic traffic sampling. It is common for a student to imitate the teacher's strengths and its shortcuts. So if the teacher is overconfident, brittle, or biased in a particular way, the student may inherit that too.

One more issue has become more visible recently: permission and contract boundaries. Distillation is technically common and legitimate when you are distilling your own models or when the provider allows it. It can become problematic if you use a third-party model's outputs to train a competing model in breach of licence or terms. Leaders need to separate the technical method from the legal right to apply it.

At its best, distillation is disciplined imitation. You use a powerful teacher to generate training signal that would otherwise be expensive to obtain, then compress that signal into a student that is cheaper to run where it counts.