Pipeline to Dashboardnihalsulaiman.com

Week 1 · Ingest anything › Day 7

Day 7. Proof 1: retail-ingest, a multi-source landing service

⏱ 4h · build 150m · README 60m · review 30m🏷 retail (+ all sources)Before: Days 1 to 6

By the end of today you can

  • Ship a GitHub-ready ingestion project with tests and a README
  • Show a manifest-driven daily summary as your first "dashboard"
  • Tell the project as a two-minute interview story

1. What you're shipping

One repo, retail-ingest, that lands all four sources into a raw zone with one manifest and one command:

uv run p2d-ingest --business-date 2026-09-14     # or --since 2026-09-01

Acceptance criteria

# Requirement How you prove it
1 Email, folder, SFTP and API sources land into raw/<source>/business_date=…/ Tree screenshot in README
2 Rerunning the same day lands nothing new test_idempotent_rerun
3 Duplicate attachments and resent POS files are detected Manifest query in README
4 Partial files and unknown senders are held back with a reason _rejected/ listing + test
5 Secrets only from env; .env.example present gitleaks passes in pre-commit
6 A completeness report: expected vs received per source per day HTML or Markdown output
7 README: problem, architecture diagram, how to run in 5 minutes, design decisions, what I'd do next Read it as a stranger

2. The completeness report (your first dashboard)

Build an expected-vs-received matrix: rows = sources/stores, columns = business dates, cell = file count or ✗. Generate it from the manifest with DuckDB and render a small HTML table with conditional colours, or a Matplotlib heatmap saved as PNG. It immediately shows R107's four-day silence and the late hotel file. This is the screenshot that goes at the top of your README.

-- expected: 12 stores x every date; received: from the manifest
with dates as (select unnest(generate_series(date '2026-09-01', date '2026-09-30', interval 1 day))::date d),
stores as (select store_code from read_csv('data/reference/stores.csv')),
got as (select store, business_date, count(*) n from email_files group by all)
select s.store_code, d.d, coalesce(g.n, 0) as files
from stores s cross join dates d left join got g on g.store = s.store_code and g.business_date = d.d
order by 1, 2;
Maths and principle

Principle: completeness is a ratio of observed to expected, and "expected" must be defined before you look.

Equation: \(C = \dfrac{|\text{received} \cap \text{expected}|}{|\text{expected}|}\). For 12 stores × 30 days, \(|\text{expected}| = 360\).

Analogy: detector efficiency: counts seen over counts that should have been seen. Where it breaks: if a store closed, it isn't "missing". The expected set needs a calendar of openings and closures.

Try it: 360 expected, R107 silent 4 days, nothing else missing. \(C\)? (356/360 ≈ 98.9%.)

3. README structure that hiring managers read

  1. One-line summary and the completeness heatmap.
  2. Problem: "A 12-store grocery chain, 18 restaurants, 5 hotels and an ad agency send data by email, shared drive, SFTP and API every morning. Reports are due at 08:00."
  3. Architecture diagram (sources → landers → raw zone + manifest → report).
  4. Run it (5 commands max, including the data generator).
  5. Design decisions: copy not move, content hashing, business date rules, look-back window, why dlt or not.
  6. Tests and what they prove.
  7. What I'd do next (links to Week 2 and 3 projects).

The two-minute story

Interviewers remember stories with a problem, a decision and a number. Practise this aloud: "Daily data came from four kinds of sources and some files arrived twice or half-uploaded. I built a landing service that copies every file into an immutable raw zone keyed by content hash, so reruns are free and every report number traces back to a file. A completeness matrix showed one store silent for four days, which a 'zero sales' dashboard would have hidden." That last sentence is the one they repeat to the hiring panel.

Build checklist

  1. Add a CLI entry point ([project.scripts] p2d-ingest = "retail_ingest.cli:main") with --business-date and --since.
  2. One run_all() that calls each lander and returns a summary dict; log it as JSON.
  3. Completeness report → reports/completeness.html (and PNG for the README).
  4. Tests green: uv run pytest -q. Lint clean: ruff check ..
  5. Write the README. Ask Claude Code to review it as a hiring manager who has 2 minutes, then fix what it says is unclear.
  6. Keep the repo local until you've chosen where to publish (you'll get the GitHub step in the Projects page).

Check yourself

1. What makes the completeness report more honest than a sales dashboard for spotting a silent store?

2. Which README section most often decides whether a reviewer keeps reading?

My notes