How it works
A data pipeline is a chain of steps that moves information from a source location to a destination consumed by analytics or another system. The flow starts with data extraction, continues with cleaning and transformation, and ends with loading to the destination. Each step can be run on its own and re-run without breaking the flow. Orchestration controls the order of steps, scheduling, and what happens when a step fails, ensuring the pipeline resumes at the correct point and does not wait for infinite retries. Logging for each step makes it possible to verify whether input data was received, whether output was generated, and whether the destination received the result.
Practical example
An e-commerce store sends purchase logs to a pipeline. The pipeline reads the logs, removes rows with empty fields, converts currency to Brazilian reais, aggregates the total by day, and writes the result to a database. A dashboard reads the database and shows revenue evolution. If part of the logs arrives with the currency field in the wrong format, the pipeline handles the error separately, alerts, and continues processing the rest instead of stopping report generation. The team can reproduce the issue with a specific data slice and fix the source before the next run.
What it's for
A data pipeline exists so analyses have a reliable and up to date foundation, without someone needing to copy and paste values manually. Consistent data enables safer metrics, reports, and models. Re-runs are also important: when a dashboard shows a wrong number, the team can re-run the pipeline from a specific point and fix the metric without losing data that was already processed. Without a pipeline, each analysis would depend on someone copying values manually, which is slow, not reproducible, and error prone.
Difference between X and Y
A data pipeline is the path data travels; ETL is a specific pattern within that path, with extraction, transformation, and loading. With ELT, data is loaded first and transformed at the destination.