AI glossary Operations
What is AI model evaluation?
← All glossary termsThe process of measuring how well an AI model or application performs: accuracy, relevance, safety, bias, cost and other criteria.
Explained
Evaluation is the systematic measurement of how well an AI model or AI application performs against the criteria that matter for its use: accuracy, relevance, groundedness, safety, bias, robustness, latency and cost. For language models it combines automatic metrics, test sets with known answers, judgements by other models and review by people. Public benchmarks compare models in general; your own evaluations tell you whether a system works for your task and data. Solid evaluation is needed before a model goes to production and again every time the model, prompt or data changes.
How AI systems are evaluated
- Reference-based tests: a set of inputs with expected outputs, scored automatically for exact match, similarity or correctness.
- Model-graded evaluation: another model scores answers against a rubric, useful for open-ended text but itself in need of checking.
- Human review: experts rate a sample of outputs, the most reliable signal for quality and the most expensive.
- Adversarial testing: red teaming and stress tests to find unsafe or broken behaviour.
- Online evaluation: signals from real use in production, such as user feedback, escalations and task completion.
Benchmarks vs your own evals
Public benchmarks measure models on shared tasks so that they can be compared. They are useful to shortlist models, but they rarely reflect your documents, your users or your definition of a good answer, and models can be tuned to score well on them.
Your own evaluation set is built from real cases in your domain, with the answers your experts would accept. It is the only way to know whether a change of model, prompt or retrieval makes your system better or worse.
Evaluating agents and RAG
For retrieval-augmented systems, evaluate the parts separately: did retrieval find the right passages, and did the answer stay faithful to them? Errors in one look like errors in the other unless you measure both.
For agents, the final answer is not enough. Look at the path: which tools were called, with what arguments, whether the steps were safe and whether the agent stopped when it should have. Cost and number of steps matter too.
Frequently asked questions
What are evals in AI?
Evals are the tests used to measure an AI system's behaviour: a set of inputs, expected outcomes or scoring rules, and a way to run them repeatedly. They work like a test suite for software, but they score quality rather than only pass or fail.
How do you evaluate an LLM?
Define what a good output means for your use case, collect representative test cases, and score outputs with a mix of automatic checks, model-graded rubrics and human review. Run the same set whenever you change the model, prompt or data, and compare.
What is LLM-as-a-judge?
It is the practice of using a language model to grade the outputs of another model against a rubric. It scales better than human review, but it has its own biases, so its scores should be checked against human judgements on a sample.
What metrics are used to evaluate AI models?
It depends on the task. Common ones are accuracy, precision and recall for classification; relevance and faithfulness for retrieval and generation; plus safety, bias, robustness, latency and cost. Frameworks such as Stanford's HELM measure several of these together.
Why is evaluation important before deploying AI?
Because a model that looks good in a demo can fail on real cases, in ways that harm users or the business. Evaluation gives evidence of quality and risk before launch and a baseline to detect regressions afterwards. Risk frameworks such as NIST's AI RMF treat measurement as a core function.
Decision guides that use this term
- Single AI vendor vs multi-model strategy: should you bet on one provider?
- AI pilot vs production: what actually changes when you scale?
- Agent framework vs custom orchestration: should you adopt a framework or write your own agent loop?
- LLM evals vs human review: how should you assure the quality of enterprise AI?
- Reasoning models vs standard LLMs: when is the extra thinking worth paying for?
- Synthetic data vs real data: what should you train, tune and test your AI on?
How to put it to work
Where it shows up at Thinkia
Further reading
Related terms
Sources
- Liang et al. · Holistic Evaluation of Language Models (HELM, arXiv)
- NIST · AI Risk Management Framework (AI RMF 1.0)
- Zheng et al. · Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (arXiv)
Updated: 2026-10-21