Pipeline to Dashboardnihalsulaiman.com

Course › Week 4

Week 4. Load to the warehouse

Azure SQL, Postgres, DuckDB, ClickHouse; MERGE, watermarks, cost. Marketing agency.

Now the data goes somewhere permanent. You load into Azure SQL Database the way production teams do (staging tables, MERGE, watermarks, transactions), then load the same data into Postgres, DuckDB and ClickHouse and measure the difference.

Sector: a marketing agency (daily ad spend, clicks and conversions for three clients across four channels).

Proof project 4: ads-warehouse: one Dagster pipeline with a swappable warehouse resource that loads Azure SQL or Postgres, with idempotent upserts, a load audit table and a cost note.

Days

  1. Day 22 Choosing where the data lives 4h · learn 90m · lab 110m · ship 25m · review 15m
  2. Day 23 Azure SQL Database, hands on 4h · learn 60m · lab 145m · ship 20m · review 15m
  3. Day 24 Loading patterns that survive reruns 4h · learn 75m · lab 130m · ship 20m · review 15m
  4. Day 25 Postgres, DuckDB and ClickHouse side by side 4h · learn 60m · lab 145m · ship 20m · review 15m
  5. Day 26 T-SQL for people who run warehouses 4h · learn 80m · lab 125m · ship 20m · review 15m
  6. Day 27 Security, cost and operations in Azure 4h · learn 70m · lab 130m · ship 25m · review 15m
  7. Day 28 Proof 4: ads-warehouse, one pipeline, any warehouse 4h · build 150m · README 60m · review 30m