Skip to main content
Contact

AI glossary Models & Architecture

What is knowledge distillation in AI?

← All glossary terms
In short

Training a smaller student model to reproduce the behaviour of a larger teacher model, so it runs faster and cheaper while keeping much of its quality.

Explained

Knowledge distillation transfers what a large teacher model has learned to a smaller student model. Instead of learning only from the correct labels, the student learns from the teacher's outputs, which carry richer information, for example how likely the teacher thought each alternative answer was. The result is a compact model that is cheaper and faster to run in production, often with a modest loss in quality on the target task. It is commonly combined with quantisation and small language models (SLMs) to build efficient assistants.

How knowledge distillation works

The teacher model is run on a set of inputs, and its outputs become training targets for the student. In the method described by Hinton, Vinyals and Dean, the teacher's probabilities are softened with a temperature setting so that the student also learns which wrong answers the teacher found plausible. When true labels are available, the student is trained on both.

With language models, distillation often takes a simpler form: a large model generates high-quality answers for many examples of a task, and a smaller model is fine-tuned on them.

What it is used for

  • Reducing inference cost and latency for high-volume tasks.
  • Running models on modest hardware, on devices or on premises, where large models do not fit.
  • Building specialised small models for one task, such as classification or extraction, from a general large model.
  • Producing compact encoders, such as distilled versions of BERT, for search and embeddings.

When it makes sense

  • The task is well defined and stable, so a smaller model can specialise.
  • Volume is high enough that cost or latency per request matters.
  • There is an evaluation set to confirm that the student is good enough.
  • The licence of the teacher model allows its outputs to be used for training other models; check this before starting.

Frequently asked questions

What is the difference between knowledge distillation and fine-tuning?

Fine-tuning trains a model further on labelled data. Distillation trains a smaller model to imitate a larger one, using the larger model's outputs as targets. Distillation is often carried out through fine-tuning, with teacher-generated data instead of, or in addition to, human labels.

What is a teacher and student model?

The teacher is the large, capable model whose behaviour you want to transfer. The student is the smaller model being trained to reproduce it. The student is the one deployed in production.

Does a distilled model perform as well as the original?

Usually somewhat worse overall, but it can come close on the specific task it was distilled for. The gap depends on the task, the size difference and the quality of the training data, so it has to be measured case by case.

What is the difference between distillation and quantisation?

Distillation produces a new, smaller model. Quantisation keeps the same model but stores its numbers with lower precision so it uses less memory and runs faster. They are complementary and often used together.

Can you distil any large language model?

Technically, many can be used as teachers, but legally it depends on the model's licence and terms of use, some of which restrict using outputs to train competing models. Open-weight models with permissive licences are the simplest case.

Decision guides that use this term

Where it shows up at Thinkia

The Mesh

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert