Pipeline to Dashboard
A 7-week, senior-depth course in ETL, warehousing and BI delivery
Start Day 1What this course is
You already watched a real pipeline run: files and emails arrive every morning, Python cleans them, an orchestrator runs the jobs, the data lands in a SQL database, and Power BI turns it into reports people act on. This course teaches you to build and run that whole chain on your own, at the depth of someone who has done it for five years: not just "it works", but why each piece is shaped the way it is, what breaks at 6 a.m., and how to choose cheaper tools when the budget is small.
The core path is the method you saw:
Ingest (email, folders, shared drives, SFTP, APIs) → Clean (Python, Polars, DuckDB, Claude Code) → Orchestrate (Dagster) → Load (Azure SQL) → Model (SQL, dbt) → Deliver (Power BI, scheduled reports)
At every step you also meet the alternatives: Postgres, DuckDB, ClickHouse; Airflow, Prefect, Mage, Kestra; dlt and Airbyte; Metabase, Superset and Evidence; local open models through Ollama. By Week 7 you can design a low-cost open-source stack that a medium-sized retail, restaurant, hotel or marketing company could actually afford.
How a day works (about 4 hours)
| Block | Time | What you do |
|---|---|---|
| Learn | ~60 min | Read the lesson. Do the small calculation in each Maths and principle box. |
| Lab | ~120 min | Build the day's piece on the course's synthetic data. Type the code; don't paste it. |
| Ship | ~40 min | Commit to the week's repo with a clear message. Update the README. |
| Review | ~20 min | Quiz, then three lines in My notes: what clicked, what didn't, what to revisit. |
Day 7 of each week is a proof day: you finish a GitHub-ready project with a pipeline, tests, a dashboard and a README a hiring manager can read in two minutes. Seven weeks, seven proofs, then one capstone that ties them together.
The data
All data is synthetic and made by starter/generate_data.py. It writes the kind of mess a BI team really receives: daily sales emails from 12 retail stores (CSV and Excel attachments), POS exports from 18 fast-food restaurants on a shared drive, nightly hotel booking extracts on an SFTP folder, and paginated ad-platform JSON for a marketing agency. Faults are planted on purpose: duplicates, late and half-uploaded files, renamed columns, day-first dates, Windows-1252 text, refunds hidden as negative quantities, and a store that goes silent for four days.
python starter/generate_data.py --start 2026-09-01 --days 60 --out ./dataHow to read the "5-year lens" boxes
Juniors learn tools. People with five years learn failure modes, trade-offs and costs: what happens when a file arrives twice, who gets paged, what a query costs per month, how to explain a wrong number to a finance manager. Every lesson has at least one of these boxes. They are the part interviewers probe.
Ground rules
Synthetic or public data only. Never use any employer's names, systems, files or figures, even masked. Secrets live in .env files that are git-ignored, never in code or screenshots.
The seven weeks
Ingest anything
Landing zones, email, shared drives, APIs. Retail chain.
Week 2Clean and trust the data
Profiling, contracts, Polars/DuckDB, Claude Code, quality gates. QSR chain.
Week 3Orchestrate with Dagster
Assets, resources, partitions, sensors, checks. Hotel group.
Week 4Load to the warehouse
Azure SQL, Postgres, DuckDB, ClickHouse; MERGE, watermarks, cost. Marketing agency.
Week 5Model it in SQL
Star schemas, SCD2, advanced SQL, dbt, semantic layer. Multi-brand restaurants.
Week 6Deliver BI people trust
Power BI, DAX, open-source BI, scheduled reports, reconciliation.
Week 7Run it in production
Observability, CI/CD, tool choices, the low-cost open stack, capstone.