Skip to main content

AI glossary Operations

What is LLM observability?

← All glossary terms
In short

Tracing, metrics and logs for prompts, retrieval, tool calls, cost and quality, so you can see what an LLM application did and why.

Explained

LLM observability is the practice of capturing what happens inside an application built on language models: the prompts sent, the documents retrieved, the tools called, the responses returned, and the latency, token use and cost of each step. It applies the general idea of observability, understanding a system's internal state from its outputs, to systems whose behaviour is probabilistic. Quality scores and user feedback are recorded alongside the technical signals. It closes the loop between experimentation and reliable production agents, because problems that only appear with real users become visible and traceable.

What it captures

  • Traces: the full path of a request, from the user's input through retrieval, model calls and tool calls to the final answer.
  • Inputs and outputs of each step, including the prompt actually sent and the context retrieved.
  • Latency, token counts and cost, per call and per conversation.
  • Errors, timeouts, retries and fallbacks between models.
  • Quality signals: automatic evaluation scores, guardrail triggers and user feedback.

Monitoring vs observability

Monitoring tells you that something is wrong: error rates are up, latency is high, the bill has grown. Observability lets you find out why, by following one request through every step and seeing what the model received and produced.

With LLMs the distinction matters more than usual. A response can be fast, error-free and still wrong. Only by looking at the context, the prompt and the output together can you tell whether the problem was retrieval, instructions, the model or a tool.

Getting it right

  • Instrument from the first prototype, not after launch: the first real users are where the surprises are.
  • Use open standards such as OpenTelemetry so that traces are not locked into one vendor.
  • Decide what you log: prompts and outputs can contain personal or confidential data, so apply masking, retention limits and access control.
  • Link traces to evaluations, so that failing cases become test cases.
  • Track cost per task, not only per call, to see what each use case really costs.

Frequently asked questions

Why do LLM applications need observability?

Because their behaviour is not fully predictable from the code. The same input can produce different outputs, and an answer can look fine while being wrong. Observability gives you the evidence to debug failures, control cost and show what the system did.

What is a trace in LLM observability?

A trace is the record of a single request as it moves through the system. It is made of spans, one for each step such as a retrieval, a model call or a tool call, with their inputs, outputs, timing and metadata.

What metrics should you track for an LLM in production?

Latency, error rate, token use and cost are the basics. On top of those, track quality signals such as evaluation scores, groundedness of answers, guardrail triggers and user feedback. Cost and quality per use case are usually more useful than global averages.

Does OpenTelemetry support generative AI?

Yes. OpenTelemetry has semantic conventions for generative AI that standardise attribute names for model requests, responses and token usage. The conventions were still marked as in development at the time of writing, so names can change.

Is LLM observability a privacy risk?

It can be, because logs may hold personal or confidential data from prompts and documents. Mask sensitive fields, limit retention, restrict access and make sure logging is covered by your data protection assessment.

Decision guides that use this term

How to put it to work

Where it shows up at Thinkia

Products

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert