Stack map: core path and alternatives
The core path is the method you're mastering. Each row lists open-source or cheaper options and when they win. Weeks in brackets show where each is taught.
| Stage | Core path | Open-source / low-cost options | Managed / enterprise options | Choose the alternative when… |
|---|---|---|---|---|
| Email ingest | Microsoft Graph (M365) or IMAP + Python email [W1] |
imap-tools, dlt |
Power Automate, Logic Apps | Non-Microsoft mail → IMAP/Gmail API; low-code team → Power Automate |
| Files / drives | Graph (SharePoint/OneDrive), paramiko (SFTP), fsspec [W1] | rclone, MinIO, dlt filesystem |
AzCopy, ADF copy, Fivetran | Many backends → rclone; Azure-only → ADF |
| APIs / DBs | requests + backoff; dlt [W1] | dlt, Airbyte OSS, Meltano, Debezium (CDC) | Fivetran, Airbyte Cloud, ADF | Many SaaS connectors → Airbyte/Fivetran; CDC → Debezium |
| Clean | Polars, DuckDB, pandera, Claude Code [W2] | pandas, Great Expectations, Soda Core, Ollama models | Claude API, Fabric dataflows | Unstructured text → LLM cascade; YAML checks → Soda |
| Orchestrate | Dagster [W3] | Airflow, Prefect, Mage, Kestra, cron | Dagster+, Astronomer, MWAA, ADF/Fabric pipelines | Existing Airflow estate → stay; low-code Microsoft → ADF |
| Load / warehouse | Azure SQL Database [W4] | Postgres, DuckDB, ClickHouse | Fabric Warehouse, Snowflake, BigQuery, Databricks SQL | No Microsoft → Postgres; billions of events → ClickHouse; many teams/elastic → cloud MPP |
| Lake storage | Folder / Azure Blob (ADLS) + Parquet [W1, W7] | MinIO, Iceberg, Delta Lake | OneLake, S3, GCS | Multi-engine lake → Iceberg/Delta |
| Model | SQL + dbt Core [W5] | SQLMesh, stored procedures | dbt Cloud, Dataform | Strong env/plan needs → SQLMesh |
| Semantic layer | Power BI semantic model + metric dictionary [W5–6] | Cube, MetricFlow | dbt Semantic Layer, LookML | Several BI tools must agree → separate layer |
| BI | Power BI [W6] | Metabase, Superset, Evidence, Lightdash, Streamlit | Tableau, Looker | Many viewers without M365 → Metabase; SQL-first data team → Superset; reports as code → Evidence |
| Delivery | Power BI subscriptions; Python email/PDF [W6] | Metabase subscriptions, Jinja + WeasyPrint | Paginated reports, Power Automate | Pixel-perfect statements → paginated reports |
| Observability | Dagster UI, audit + DQ tables [W3, W7] | Grafana, OpenLineage/Marquez | Monte Carlo, Datadog | Many pipelines/teams → dedicated tools |
| CI/CD | GitHub Actions + Docker [W7] | GitLab CI, act | Azure DevOps | Company standard decides |
| Secrets / identity | Entra managed identity, Key Vault [W4] | Docker secrets, SOPS, Vault | AWS Secrets Manager, GCP Secret Manager | Not on Azure → platform's own |
Three reference stacks
A. Microsoft medium company (M365, Power BI users): Graph ingest → Dagster on a VM (or ADF) → Azure SQL (serverless GP, columnstore facts) → dbt-sqlserver → Power BI Pro, RLS, subscriptions. Identities: managed identity + Key Vault.
B. Open-source low-cost (no licences, 1–3 data people): dlt + Python landers → Dagster OSS → MinIO/Blob raw Parquet → Postgres (+ DuckDB) → dbt-postgres → Metabase/Superset + Python email reports; Ollama for documents. One or two VMs, Docker Compose. (Week 7, Day 46.)
C. Scale-up (many teams, large event data): Airbyte/Fivetran → Airflow or Dagster+ → Snowflake/BigQuery/Databricks or ClickHouse for events → dbt → Power BI/Looker/Tableau; Iceberg/Delta lake.
All three share the same ideas: immutable raw, idempotent loads, business dates, contracts and quarantine, star schemas, one metric dictionary, missing ≠ zero. That's what makes you portable across them.