Pipeline to Dashboardnihalsulaiman.com

Course › Week 1

Week 1. Ingest anything

Landing zones, email, shared drives, APIs. Retail chain.

Every pipeline starts with a promise you don't control: someone else will send you data. This week you build the front door of a data platform: a landing zone that accepts files from email, shared folders, SFTP and APIs, records exactly what arrived and when, and never loses or double-counts a file.

Sector: retail chain (12 grocery stores emailing daily sales) plus first contact with the QSR, hotel and ad data.

Proof project 1: retail-ingest: a multi-source ingestor that pulls email attachments, folder drops and API pages into an immutable raw zone (Parquet) with a manifest table, hashes, and a rerun-safe design.

You should already know You'll leave knowing
Basic Python, basic SQL SELECT ETL vs ELT, layers (raw/staging/marts), idempotency, manifests, IMAP vs Microsoft Graph, SFTP, paginated APIs, secrets handling

Days

  1. Day 1 The shape of a data platform 4h · learn 75m · lab 120m · ship 30m · review 15m
  2. Day 2 Engineering setup that survives a real job 4h · learn 60m · lab 135m · ship 30m · review 15m
  3. Day 3 Landing files from folders and shared drives 4h · learn 60m · lab 140m · ship 25m · review 15m
  4. Day 4 Extracting data from email 4h · learn 70m · lab 130m · ship 25m · review 15m
  5. Day 5 SharePoint, OneDrive, Google Drive, SFTP and object storage 4h · learn 60m · lab 140m · ship 25m · review 15m
  6. Day 6 APIs and source databases 4h · learn 70m · lab 130m · ship 25m · review 15m
  7. Day 7 Proof 1: retail-ingest, a multi-source landing service 4h · build 150m · README 60m · review 30m