Day 20. Asset checks, testing and deploying Dagster

⏱ 4h · learn 60m · lab 140m · ship 25m · review 15m🏷 hospitalityBefore: Day 19

By the end of today you can

Move Week 2 DQ checks into Dagster asset checks, some of them blocking
Use freshness checks for "data must be newer than X"
Run Dagster OSS in Docker with Postgres storage
Choose between self-hosted Dagster and Dagster+ with costs in mind

1. Why this matters

Your DQ checks from Day 13 run outside the orchestrator. Moving them into Dagster means a failed check can stop bad data from flowing downstream, show up next to the asset it guards, and alert through the same channel as failures.

2. Asset checks

@dg.asset_check(asset=stg_bookings, blocking=True, description="booking_id unique after dedup")
def bookings_unique(context) -> dg.AssetCheckResult:
    df = pl.read_parquet("data/staging/hotel/stg_bookings.parquet")
    dups = df.height - df["booking_id"].n_unique()
    return dg.AssetCheckResult(passed=dups == 0, metadata={"duplicates": dups})

@dg.asset_check(asset=stg_bookings)
def rate_in_range(context) -> dg.AssetCheckResult:
    df = pl.read_parquet("data/staging/hotel/stg_bookings.parquet")
    bad = df.filter((pl.col("rate_amount") < 40) | (pl.col("rate_amount") > 1500)).height
    return dg.AssetCheckResult(passed=bad / df.height < 0.001, severity=dg.AssetCheckSeverity.WARN,
                               metadata={"out_of_range": bad})

blocking=True means downstream assets in the same run don't materialise if the check fails. Use it for checks whose failure means "the numbers would be wrong" (duplicates, missing partition), and WARN severity for "worth a look".

Freshness: Dagster provides freshness checks (build_last_update_freshness_checks and newer freshness policies depending on version) that fail when an asset hasn't been updated within a window, e.g. "hotel staging must be updated by 07:30 daily".

3. Testing strategy

Layer	What	Tool
Unit	Pure core functions with nasty fixtures	pytest
Asset	Asset function with fake inputs	pytest, direct call
Graph	A few assets wired with local resources	`dg.materialize`
Definitions load	The code location imports cleanly	`dg.Definitions.validate_loadable(defs)` in a test
Data	Checks on real outputs	asset checks

4. Deploying Dagster OSS

Dagster has three long-running parts: the webserver (UI), the daemon (schedules, sensors, run queue), and your code location (a gRPC server loading your definitions). Storage for run history goes in Postgres in production (SQLite is fine locally).

# docker-compose.yml (sketch)
services:
  postgres:   { image: postgres:16, environment: { POSTGRES_USER: dagster, POSTGRES_PASSWORD: dagster, POSTGRES_DB: dagster } }
  code:       { build: ., command: dagster code-server start -h 0.0.0.0 -p 4000 -m hotel_pipeline.definitions, env_file: .env }
  webserver:  { build: ., command: dagster-webserver -h 0.0.0.0 -p 3000 -w workspace.yaml, ports: ["3000:3000"], depends_on: [postgres, code] }
  daemon:     { build: ., command: dagster-daemon run -w workspace.yaml, depends_on: [postgres, code] }

dagster.yaml points run, event and schedule storage at Postgres; workspace.yaml points at the code server. The Dagster docs have a maintained Docker example; start from it.

Maths and principle

Principle: total cost of ownership = infrastructure + people time. Self-hosting trades money for time.

Equation: $\text{TCO}_{\text{month}} = C_{\text{infra}} + h \cdot w$, with $h$ hours/month of maintenance and $w$ hourly cost. A small VM (2–4 vCPU, 8 GB) is roughly tens of dollars a month; managed orchestration is priced per usage or seat. Compare at your $h$.

Analogy: owning vs renting. Where it breaks: hidden $h$, like upgrades, security patches, and the 2 a.m. "the daemon stopped" call.

Try it: VM $40/month, 4 h/month upkeep at $50/h vs a managed plan at $250/month with 0.5 h upkeep. Which is cheaper? (Self-host: 40 + 200 = $240; managed: 250 + 25 = $275. Close: reliability and team skills decide.)

The 5-year lens

For a medium-sized company with one or two data people, a single VM running Docker Compose (Dagster + Postgres) with nightly backups of the Dagster database and the code in git is a perfectly respectable production setup. Write it down: how to restart, how to upgrade, where logs are, who to call. Most outages in small teams are "nobody knew how to restart it".

Lab

Add 5 asset checks to the hotel graph (uniqueness blocking, rate range warn, null arrival blocking, partition row count vs robust median warn, referential hotel_id blocking).
Seed a fault (duplicate a file's rows into staging) and confirm the blocking check stops otb_by_stay_date.
Add a freshness check for staging. Let it go stale and see it fail.
Write the test layers table into your repo with at least one test per layer.
Run the whole thing with Docker Compose and Postgres storage. Restart the containers and confirm run history survives.

Ship

feat(dagster): asset checks, freshness, layered tests, docker-compose deploy.