AI glossary Data & Retrieval
What is a data pipeline?
← All glossary termsAn automated sequence of steps that collects data from its sources, cleans and transforms it, and delivers it to the models, applications or analytics that use it.
Explained
A data pipeline is an automated workflow that ingests data from sources such as databases, files, applications or APIs, transforms it and moves it to a destination such as a data warehouse, a feature store, a vector index or an application. Each step is defined in code or configuration and runs on a schedule or whenever new data arrives. Pipelines replace manual extracts and spreadsheets with a repeatable process that can be monitored and audited. For AI they are critical: a model or RAG system is only as current and reliable as the data that reaches it.
Stages of a data pipeline
- Ingestion: read data from the source systems, in full or only what has changed.
- Validation: check formats, required fields and expected ranges, and quarantine what fails.
- Transformation: clean, join, enrich and reshape the data for its use.
- Delivery: load it into the destination, such as a warehouse, a lakehouse, a feature store or a vector index.
- Monitoring: track freshness, volumes and errors, and alert when something breaks.
Batch or streaming?
Batch pipelines process data in blocks at set intervals, for example every night. They are simpler and cheaper, and suit reporting and model training. Streaming pipelines process each event as it happens, which suits fraud detection, real-time recommendations or operational alerts.
Many architectures combine both: streaming for what needs to be immediate, batch for heavy processing and history.
Pipelines for AI
AI adds its own pipelines on top of the classic ones. Training pipelines prepare labelled datasets and features. RAG pipelines split documents into chunks, generate embeddings and keep the vector index in sync with the source, including deletions and permission changes.
The usual failure in production is not the model but stale or broken data. Treating pipelines as products, with owners, tests and monitoring, prevents most of it.
Frequently asked questions
What is the difference between a data pipeline and ETL?
ETL (extract, transform, load) is one type of data pipeline, where data is transformed before it is loaded into the destination. Data pipeline is the broader term: it also covers ELT, where data is loaded first and transformed later, streaming flows and pipelines that feed models or applications.
What is an example of a data pipeline?
A nightly process that takes the day's sales from the point-of-sale system, removes duplicates, adds product and store data and loads it into the data warehouse is a typical example. Another is a flow that turns new documents from a repository into embeddings for a RAG assistant.
Which tools are used to build data pipelines?
Common choices include orchestrators such as Apache Airflow, transformation tools such as dbt, streaming platforms such as Apache Kafka, and the managed services of the main cloud providers. The right mix depends on data volume, latency needs and the team's skills.
Why do data pipelines matter for AI?
Because models learn from and answer with the data they receive. If a pipeline delivers stale, incomplete or wrong data, the model's outputs will be wrong too, however good the model is. Reliable pipelines are a precondition for AI in production.
What is a RAG ingestion pipeline?
It is the pipeline that prepares documents for retrieval-augmented generation: it reads them, splits them into chunks, generates embeddings and stores them in a vector index with their metadata and permissions. It must also update or remove entries when the source documents change.
Decision guides that use this term
How to put it to work
Where it shows up at Thinkia
Further reading
Related terms
Sources
- Microsoft · Big data architectures (Azure Architecture Center)
- Microsoft · Choose a stream processing technology (Azure Architecture Center)
- Apache Airflow · Core concepts (DAGs, tasks, operators)
Updated: 2026-10-21