Day 15. What an orchestrator really does

⏱ 4h · learn 80m · lab 120m · ship 25m · review 15m🏷 hospitalityBefore: Week 2

By the end of today you can

Explain DAGs, task-based vs asset-based orchestration and why Dagster chose assets
Reason about at-least-once execution and why every step must be idempotent
Separate scheduling time from business time
Say what belongs in the orchestrator and what doesn't

1. Why this matters

Cron can run a script at 06:00. It can't tell you the script failed, rerun only the failed part, wait for a late file, backfill last month after a bug fix, or show which dashboard depends on which file. An orchestrator does all of that. Choosing and using one well is one of the clearest lines between junior and senior data work.

2. The DAG

A pipeline is a directed acyclic graph: nodes are steps (or data assets), edges are "must happen before". Acyclic because a step can't depend on its own output.

raw_hotel_files ──► stg_bookings ──► fct_bookings_daily ──► revenue_report
reference_hotels ──────────────────┘

Task-based orchestrators (Airflow, classic Prefect, cron) describe what to run. Asset-based orchestrators (Dagster; Airflow 3 added "assets" too) describe what data should exist, and the steps that produce it. With assets the orchestrator knows lineage, freshness and which downstream tables are stale after an upstream change.

Maths and principle

Principle: a DAG has at least one topological order, and any valid order produces the same results if steps are pure.

Equation: for edges \(u \to v\), a topological order \(\sigma\) satisfies \(\sigma(u) < \sigma(v)\). Steps with no path between them can run in parallel; the minimum wall time is the length of the critical path \(\max_{\text{paths } P} \sum_{v \in P} t_v\).

Analogy: a partial order, like prerequisites in a degree. Where it breaks: hidden dependencies (two steps writing the same table) aren't edges, so the orchestrator can run them in parallel and corrupt data. Make every dependency explicit.

Try it: extract 10 min, three parallel cleans 6/9/4 min, then a model step 5 min. Critical path? (10 + 9 + 5 = 24 min.)

3. At-least-once means idempotent

Orchestrators retry. Machines restart mid-run. A run may execute twice for the same day. So every step must be safe to repeat: land by hash (Day 3), replace a partition instead of appending, upsert by key (Week 4). If a step sends an email, guard it with a "sent" record.

4. Two clocks

Scheduling time: when the run starts (06:15 on 15 Sept).
Business time: which data it processes (business date 14 Sept).

Never compute business time from now() inside a step. The orchestrator passes it in (a partition key). Then a backfill for 1 Aug runs exactly the same code as today's run.

5. What belongs where

In the orchestrator	In your code / libraries	In the warehouse
Schedules, sensors, retries, partitions, lineage, alerts, run history	Parsing, cleaning rules, API clients (testable without the orchestrator)	Business logic in SQL/dbt, constraints

The 5-year lens

Keep the orchestrator thin. Your Week 1–2 functions should run from a plain test without Dagster installed; Dagster wraps them. Teams that put logic inside orchestrator-specific callbacks can't test it, can't migrate (Airflow → Dagster migrations are common), and can't run it locally. In an interview, "my assets call pure functions from my package" is a strong signal.

The orchestrator landscape (details in Week 7)

Dagster (asset-based, strong local dev and testing, this course's core). Airflow (the incumbent, huge ecosystem, Airflow 3 modernised it). Prefect (Pythonic flows, easy start). Mage (notebook-like blocks). Kestra (YAML, event-driven). Azure Data Factory / Fabric (low-code, Microsoft). cron + scripts (fine for 3 jobs, painful at 30).

Lab: design before code

Draw the asset graph for the hotel group: raw PMS files, hotel reference, staged bookings (latest version), daily on-the-books by stay date, a revenue report. Mark partitions (by business date) and which nodes read from which.
For each node, write: inputs, output location, idempotency strategy (replace partition / upsert / hash), expected run time, owner.
Write a 1-page docs/run-semantics.md: what happens if the 06:15 run fails at stg_bookings; if the hotel file arrives at 07:40; if a bug in cancellations is found 20 days later.
Install Dagster in a new repo hotel-dagster (uv add dagster dagster-webserver), run dagster dev on an empty definitions file, and click around the UI.

Ship

docs: asset graph and run semantics for the hotel pipeline.