Day 18. Partitions and backfills

⏱ 4h · learn 65m · lab 135m · ship 25m · review 15m🏷 hospitality, QSRBefore: Day 17

By the end of today you can

Partition assets by business date and process one day per run
Run and monitor a backfill after a bug fix
Handle look-back windows and late data with partitions
Use static and dynamic partitions for per-store or per-hotel work

1. Why this matters

"Rerun last month" is a weekly request in real teams: a rule changed, a source re-sent history, a bug was fixed. With partitions it's a few clicks and costs exactly the compute it needs. Without them, it's a frightening one-off script.

2. Daily partitions

import dagster as dg
daily = dg.DailyPartitionsDefinition(start_date="2026-09-01", timezone="America/Toronto", end_offset=0)

@dg.asset(partitions_def=daily)
def stg_qsr_pos(context: dg.AssetExecutionContext, raw: RawZone) -> dg.MaterializeResult:
    bdate = context.partition_key                     # "2026-09-14": business date, passed in
    df = qsr.clean_day(raw, bdate)
    path = f"data/staging/qsr_pos/business_date={bdate}/part.parquet"
    df.write_parquet(path)                            # REPLACE this partition: idempotent
    return dg.MaterializeResult(metadata={"rows": len(df)})

Each partition is materialised and tracked separately. The UI shows a bar of green/red/missing days.

The QSR business day ends at 04:59

Business date 14 Sept includes files in the 14 Sept folder and orders after midnight that appear in the 15 Sept export (depending on how the POS splits files). The partition for 14 Sept therefore needs both folders. Write down which raw inputs each partition reads; that's its dependency window.

3. Backfills

In the UI: select the asset → Materialize → pick a range → launch a backfill. Dagster queues one run per partition (or a single run for the range, if your asset supports BackfillPolicy.single_run() and handles context.partition_key_range). Watch concurrency: 60 parallel runs hitting an SFTP server is a denial of service. Limit with run concurrency settings or tag-based limits.

Maths and principle

Principle: partitioning turns one big function into many small independent ones, so you only recompute what's affected.

Equation: if \(T = \bigsqcup_d T_d\) (disjoint by day) and \(f(T) = \bigsqcup_d f(T_d)\) (the transform works day-by-day), then fixing a bug affecting days \(D' \subset D\) costs \(\sum_{d \in D'} c(f, T_d)\), not \(c(f, T)\).

Analogy: a block-diagonal matrix: each block can be solved on its own. Where it breaks: transforms that need several days (7-day rolling averages, "latest version" across days) aren't block-diagonal. They need a window of upstream partitions, a TimeWindowPartitionMapping, or an unpartitioned downstream asset.

Try it: 365 partitions × 40 s each. Bug affected 20 days. Backfill time with 4 concurrent runs? (20 × 40 / 4 = 200 s.)

4. Upstream windows and look-backs

@dg.asset(partitions_def=daily,
          ins={"stg_ads": dg.AssetIn(partition_mapping=dg.TimeWindowPartitionMapping(start_offset=-7, end_offset=0))})
def ads_rolling_7d(stg_ads: pl.DataFrame) -> pl.DataFrame:
    ...  # receives 8 days of upstream partitions

For the ads look-back (Day 6), the daily schedule can request the last 7 partitions each morning, so recent days are refreshed as conversions arrive.

5. Static, dynamic and multi-partitions

Static: dg.StaticPartitionsDefinition(["H1","H2","H3","H4","H5"]) for per-hotel work.
Dynamic: partitions added at run time (a new store opens): dg.DynamicPartitionsDefinition(name="stores"), added by a sensor.
Multi: date × hotel: dg.MultiPartitionsDefinition({"date": daily, "hotel": hotels}). Powerful, but the partition count multiplies. 5 hotels × 365 days = 1,825; fine. 500 stores × 3 years = 547,500; not fine. Partition by date and loop stores inside.

The 5-year lens

Partition by business date, not file arrival date: backfills then mean "recompute what we believe about 14 Sept", which is what the business asks. And make every partitioned write a replace (delete-insert or overwrite the folder), never an append. An append-mode partitioned asset is a double-count waiting for its first retry.

Lab

Partition stg_qsr_pos and stg_hotel_bookings_daily (bookings created on that business date) by day.
Materialise 10 days. Then change a cleaning rule (e.g. treat qty > 20 as suspicious) and backfill all 60 days. Record time and runs in the README.
Implement the QSR dependency window: partition 14 Sept reads both folders and filters by business date. Test the 04:59/05:00 boundary across folders.
Build ads_daily partitioned, plus a schedule (tomorrow's lesson formalises it) that requests the last 7 partitions.
Make otb_by_stay_date unpartitioned downstream of partitioned staging, and explain in docs/run-semantics.md why.

Ship

feat(dagster): daily partitions, backfill demo, look-back window.