| Transformation | Purpose | |---|---| | | Reads from a dataset (JSON, Parquet, CSV). | | Filter | Removes rows based on condition (e.g., Price > 100 ). | | Derived Column | Creates new columns or modifies existing ones (e.g., Total = Price * Quantity ). | | Aggregate | Group by and compute sum, avg, min, max. | | Join | Combines two streams (Inner, Left Outer, Full Outer). | | Sink | Writes transformed data to destination (Delta Lake, SQL, ADLS). |
At its core, Azure Data Factory is a managed, serverless platform designed for complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects. It provides a visual environment to construct pipelines that ingest raw data from various sources and refine it into actionable business insights. javatpoint azure data factory