Azure Data Factory (ADF) is a data integration and orchestration service. A pipeline describes a workflow that connects sources, activities, and destinations; ADF coordinates execution and exposes run details for monitoring.
The core building blocks
- Pipelines group activities into a workflow and express dependencies between steps.
- Activities perform work such as copying data, running a query, or invoking another process.
- Linked services define connection information for a data store or compute service.
- Datasets describe the structure or location of the data an activity reads or writes.
- Integration Runtime provides the execution and connectivity environment for data movement and dispatch.
Triggers, parameters, and variables
Triggers start pipelines on a schedule, in response to an event, or on demand. Parameters pass values into a run, such as a source path or processing date. Variables hold values during a pipeline run when later activities need them. Parameters make reusable pipeline designs possible without copying the workflow.
Monitoring and recovery
- Inspect the pipeline run and then the failed activity for error details and duration.
- Track rows read, written, and rejected where the connector exposes those measures.
- Retry only failures that are likely transient and keep retry limits bounded.
- Advance an incremental watermark only after the target write and validation succeed.
A beginner's practice flow
Sketch a scheduled file copy: a trigger starts the pipeline, a parameter supplies the date, a linked service connects to storage, a dataset describes the file, and a copy activity writes to a landing location. Then add a validation step and review the run in monitoring.