What is metadata, and why does it matter for AI?

Explained

Metadata is data about data: the information that describes a document, a dataset or a record, such as who created it, when, what it is about, what format it has and who may see it. In search and RAG systems, metadata lets you filter results by date, department, product or permissions, and rank recent or authoritative sources higher. In data and AI governance it records lineage, ownership and licensing. Poor or missing metadata is one of the most common reasons enterprise assistants return outdated or unauthorised content.

Types of metadata

• Descriptive: title, author, subject, keywords, language.
• Structural: how a document or dataset is organised, such as sections, pages or table schema.
• Administrative: owner, creation and update dates, retention, licence.
• Access: who can read or edit, sensitivity labels.
• Provenance and lineage: where the data came from and how it was transformed.

Why it matters in RAG and search

Semantic search finds passages that are similar in meaning, but it does not know whether a document is current, approved or visible to the person asking. Metadata fills that gap.

Typical uses are filtering by permissions before retrieval, preferring the latest version of a policy, restricting results to one country or product line, and showing the user the source and date of each passage. Many vector databases store metadata alongside each vector precisely for this.

Good practice

• Agree a small, shared schema before indexing, rather than collecting everything.
• Inherit access permissions from the source system instead of copying them by hand.
• Extract metadata automatically where you can, and check it on a sample.
• Keep dates and versions reliable: freshness is often what users notice first.
• Use standards such as Dublin Core where they fit, so metadata can be shared between systems.

Frequently asked questions

Decision guides that use this term

• Fix the data first or start with AI now: which comes first?

How to put it to work

• How to automate ESG reporting with AI
• How to improve data quality with AI
• How to keep expert knowledge when people retire
• How to query your data in natural language

Further reading

• Smart Data Curation Is the New Scale for Enterprise AI

Related terms

• AI Governance The policies, processes and controls an organisation uses to make sure its AI is safe, lawful, ethical and worth what it costs.
• RAG A technique that retrieves relevant passages from a knowledge base and gives them to a language model as context, so its answer rests on those sources.
• Big Data Data sets so large, fast-moving or varied that traditional databases and tools cannot store or process them efficiently; a key input for AI.
• Context Engineering The practice of deciding what information a model sees at each step (instructions, retrieved documents, tool results, memory) and fitting it into the context window.
• Corpus A collection of documents or texts used to train a model or to serve as the knowledge base an AI system retrieves from, as in RAG.
• Data Lake A central repository that stores structured and unstructured data in its raw form, as a source for analytics, machine learning and AI.