Skip to main content

AI glossary Operations

What is inference in AI?

← All glossary terms
In short

The phase in which a trained model is used to produce outputs, such as a prediction, a classification or generated text, from new inputs.

Explained

Inference is the moment a trained model is put to work: it receives a new input and produces an output, such as a forecast, a label, an image or an answer from a language model. Training happens once or from time to time; inference happens every time someone uses the system. That is why, in production, most of the running cost and the user's experience of speed depend on inference. Optimising it means balancing latency, throughput, cost and quality through choices of model size, hardware and serving techniques.

Training vs inference

Training adjusts a model's parameters using large datasets and a lot of compute. Inference keeps those parameters fixed and only runs the model forward on each new input.

A single inference call is far cheaper than training, but it is repeated for every request, every user and every day. For a widely used assistant, the accumulated cost of inference can exceed the cost of building it.

What drives latency and cost

  • Model size: more parameters mean more compute per token or prediction.
  • Input and output length: for language models, longer prompts and answers cost more.
  • Hardware: GPUs, TPUs, other accelerators or CPUs, in the cloud or on a device.
  • Concurrency: how many requests are served at once and how they are batched.
  • Memory: for LLMs, the key-value (KV) cache that stores the context while generating.

How teams optimise it

  • Choose the smallest model that meets the quality bar, and route only hard cases to a larger one.
  • Quantise or distil models so they need less memory and compute.
  • Use serving engines that batch requests and manage memory efficiently, such as vLLM.
  • Cache repeated prompts and results.
  • Run inference at the edge when latency or data location requires it.
  • Monitor latency, cost per request and quality together, not one at a time.

Frequently asked questions

What is inference in machine learning?

It is using a trained model to make predictions on new data. For example, a fraud model scoring a new transaction, or a language model answering a question, are both doing inference.

What is the difference between training and inference?

Training is when the model learns from data and its parameters change. Inference is when the finished model is used and its parameters stay fixed. Training is occasional and heavy; inference is continuous and has to be fast.

Why is AI inference expensive?

Because large models need a lot of compute and memory for each request, and requests add up. For language models the cost also grows with the number of tokens processed and generated, so long prompts and long answers are more expensive.

What is inference latency?

It is the time between sending a request and getting the result. For chat and voice interfaces, the time to the first token and the speed of generation both matter, because they decide whether the interaction feels natural.

Where does inference run?

It can run in the cloud through an API, on your own servers, or on devices such as phones, cameras and industrial equipment. The choice depends on latency, cost, data protection and the size of the model.

Decision guides that use this term

Where it shows up at Thinkia

Products

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert