Pipeline to Dashboardnihalsulaiman.com

Week 7 · Run it in production › Day 43

Day 43. Observability and incident response

⏱ 4h · learn 70m · lab 135m · ship 20m · review 15m🏷 allBefore: Week 6

By the end of today you can

  • Define freshness and completeness SLOs per report and measure them
  • Build a pipeline health dashboard from run history, audit and DQ tables
  • Triage an incident with an evidence ladder and a clear verdict
  • Write a blameless post-incident review

1. Why this matters

Once a platform has users, your job is half building and half keeping promises: "the QSR report is in inboxes by 08:00 with complete data on 98% of days". Observability is how you know whether you're keeping them; incident response is what you do when you aren't.

2. SLOs for data

Report Freshness SLO Completeness SLO Measured from
QSR daily email Sent by 08:00 on 98% of days ≥ 17 of 18 restaurants, or banner ops.report_sent, manifest
Hotel revenue page Data as of ≤ 07:30 on 95% of days All 5 hotels ops.load_audit
Ads dashboard Updated by 12:00 daily All campaigns with spend DQ results

An error budget is the allowed misses: 98% over 30 days = 0.6 days. When the budget is spent, reliability work beats new features.

3. The health dashboard

Build it from tables you already have: Dagster run history (via its GraphQL API or Postgres storage), ops.load_audit, dq_results, the manifest, ops.report_sent. Panels: runs by status per day, duration trend per asset (p50/p95), freshness per source vs SLO, DQ failures by family, error budget remaining. Grafana or Metabase both work.

Maths and principle

Principle: monitor the tail, not the mean. A pipeline whose average finish is 06:40 can still miss 08:00 often.

Equation: SLO compliance \(= \frac{1}{n}\sum_{i=1}^{n} \mathbb{1}[t_i \le \text{deadline}]\); track p95 of \(t_i\) with a rolling window. Error budget remaining \(= (1 - \text{SLO}) \cdot n - \text{misses}\).

Analogy: reliability engineering in physics labs: a detector that's up 99% of the time but down during the one event that matters failed its job. Where it breaks: small \(n\). Over 30 days, one miss is 3.3%, so judge SLOs over longer windows too.

Try it: SLO 98% over 30 days, 1 miss so far. Budget left? (0.6 − 1 = −0.4: already over budget.)

4. Triage with an evidence ladder

When an alert fires or a user complains, climb from the cheapest evidence to the most expensive, and stop when the evidence settles it:

  1. Is it real? Check the alert's own data (maybe the check was unavailable, not failed).
  2. Where in the chain? Manifest (did the file arrive?) → run log (did the asset run?) → audit (rows loaded?) → DQ (quarantined?) → mart (query) → report (refresh time).
  3. What changed? Source file shape, code deploys (git log), config, volumes.
  4. Verdict, one of: source issue (contact owner), pipeline bug (fix + test), expected behaviour (explain, maybe improve labels), transient (retry worked; watch), not yet determined (state what would settle it).

This is the method in your bi-skills-kit etl-triage skill. Use it with Claude Code on the course incidents, read-only, evidence saved.

The 5-year lens

Write a blameless post-incident review for anything that broke a promise: timeline, impact (who saw what wrong number, for how long), root cause, why it wasn't caught earlier, actions with owners and dates. Blameless means "the process let this happen", never "Sam broke it". Teams that do this get steadily more reliable; teams that don't have the same incident every quarter. Two short, honest reviews in your portfolio repo are rare and impressive.

Lab

  1. Define SLOs for your three reports in docs/slo.md.
  2. Build ops.report_sent and make the email asset write to it. Compute SLO compliance for the last 30 simulated days (run the pipeline over 30 days with backfills; inject 2 late hotel files and 1 broken QSR file by editing the generator's seeds or files).
  3. Build the health dashboard in Metabase or Grafana.
  4. Run a triage on each injected incident with the evidence ladder; use Claude Code with the etl-triage skill in read-only mode. Save triage/<date>-<slug>/report.md with evidence.
  5. Write one blameless review for the worst incident.

Ship

Start the capstone repo daily-data-platform. Commit docs+feat(ops): SLOs, health dashboard, triage reports, incident review.

Check yourself

1. A freshness check shows "unavailable" because the manifest DB was down. Verdict for the data?

2. Why track p95 finish time instead of the average?

3. What makes a post-incident review "blameless"?

My notes