Pipeline to Dashboardnihalsulaiman.com

Week 5 · Model it in SQL › Day 31

Day 31. SQL the way senior analysts write it

⏱ 4h · learn 60m · lab 150m · ship 15m · review 15m🏷 all fourBefore: Day 30

By the end of today you can

  • Use window functions for running totals, ranks, period-over-period and moving averages
  • Solve gaps-and-islands (streaks, outages, stays)
  • Build date spines to show zeros honestly and cohorts for retention
  • Write readable SQL with CTEs and test it with known answers

1. Why this matters

This is your SQL drill day, closing your stated biggest gap. Every pattern below shows up in BI interviews and weekly in real work. Do them in DuckDB first (fast feedback), then run the same ones on Azure SQL to learn the T-SQL differences.

2. Window functions

with daily as (
  select restaurant_id, business_date, sum(qty * price) as revenue
  from stg_qsr_pos group by all
)
select *,
  sum(revenue) over (partition by restaurant_id order by business_date
                     rows between unbounded preceding and current row)         as running_revenue,
  avg(revenue) over (partition by restaurant_id order by business_date
                     rows between 6 preceding and current row)                 as ma7,
  revenue / nullif(lag(revenue, 7) over (partition by restaurant_id order by business_date), 0) - 1 as wow_same_weekday,
  rank() over (partition by business_date order by revenue desc)              as rank_on_day
from daily;

rows between counts rows; range between interval 6 day preceding counts time. They differ when days are missing, which is the next topic.

3. Date spines: zeros vs missing

A moving average over rows silently skips missing days. Join to a spine of all dates × all entities, then decide: missing because closed (exclude), missing because not received (flag), or genuinely zero (0).

with spine as (
  select r.restaurant_id, d.date_key as business_date
  from dim_restaurant r cross join dim_date d
  where d.date_key between date '2026-09-01' and date '2026-10-29'
)
select s.*, coalesce(t.revenue, 0) as revenue, t.revenue is null as no_data
from spine s left join daily t using (restaurant_id, business_date);

4. Gaps and islands

Find consecutive runs: a store's silent days, a guest's consecutive stays, a campaign's streak above target.

-- islands of consecutive days with no data for each store
with flagged as (
  select store_code, business_date,
         business_date - cast(row_number() over (partition by store_code order by business_date) as int) as grp
  from spine_with_flags where no_data
)
select store_code, min(business_date) as gap_start, max(business_date) as gap_end, count(*) as days
from flagged group by store_code, grp order by days desc;
Maths and principle

Principle: in a run of consecutive integers, value minus rank is constant. That constant labels the island.

Equation: for sorted days \(d_1 < d_2 < \dots\) with ranks \(r_i\), \(g_i = d_i - r_i\). If \(d_{i+1} = d_i + 1\) then \(g_{i+1} = g_i\); a gap of \(k\) days makes \(g\) jump by \(k-1\).

Analogy: subtracting a linear trend: what's left is flat inside each run and steps between runs. Where it breaks: duplicate days (rank them with dense_rank or dedup first).

Try it: days 3, 4, 5, 9, 10. Ranks 1–5. Groups? (2, 2, 2, 5, 5: two islands.)

5. Cohorts

Retention by first-purchase month for a loyalty programme (retail) or repeat guests (hotels):

with firsts as (select customer_id, date_trunc('month', min(order_date)) as cohort from orders group by 1),
activity as (select distinct customer_id, date_trunc('month', order_date) as m from orders)
select cohort, datediff('month', cohort, m) as month_n, count(*) as active
from firsts join activity using (customer_id)
group by 1, 2 order by 1, 2;

6. T-SQL differences to remember

DuckDB / Postgres T-SQL
limit 10 select top (10) or offset 0 rows fetch next 10 rows only
date_trunc('month', d) datetrunc(month, d) (SQL Server 2022 / Azure SQL) or datefromparts(year(d), month(d), 1)
d - interval 7 day dateadd(day, -7, d)
qualify wrap in a CTE and filter rn = 1
group by all list the columns
string_agg(x, ',') string_agg(x, ',') (same)
generate_series generate_series(1, n) on SQL Server 2022+/Azure SQL, or a numbers table

The 5-year lens

Write SQL like code: one CTE per step with a name that says what it holds (daily_revenue, spine, with_flags), no select * in final models, explicit join conditions, and known-answer tests: tiny hand-made input tables where you can compute the answer on paper. When an analyst says "your WoW number is wrong", a known-answer test settles it in a minute.

Lab: 15 problems (answers in your repo, checked by tests)

  1. Running revenue per restaurant per month, resetting each month.
  2. 7-day moving average using a time-based range window; compare with rows on a store with gaps.
  3. Same-weekday-last-week % change per restaurant.
  4. Top 3 menu items per brand per week (ties included).
  5. Each restaurant's best and worst business day, with the date.
  6. R107's silent islands (gaps and islands).
  7. Hotel: consecutive nights sold-out (occupancy OTB ≥ 98%) per hotel.
  8. Hotel: lead-time buckets (0–1, 2–7, 8–30, 31–90, 90+) share of bookings by channel.
  9. Ads: first date each campaign's cumulative spend passed $1,000.
  10. Ads: campaigns whose ROAS dropped more than 30% week-over-week.
  11. Retail: basket size distribution (lines per receipt) per store.
  12. Retail: refund rate per store per week, with zero-refund weeks shown as 0, not missing.
  13. QSR: share of revenue after midnight per brand (business-day aware).
  14. Pivot channel revenue into columns per brand per week (pivot in DuckDB, case when in T-SQL).
  15. A cohort table for synthetic loyalty customers (ask Claude Code to add a customer_id to 30% of retail receipts).

Write each as a .sql file with a known-answer test on a 10-row fixture. Run 5 of them on Azure SQL and note the dialect changes.

Ship

feat(sql): 15 analytical SQL patterns with known-answer tests (DuckDB + T-SQL).

Check yourself

1. A moving average uses ROWS BETWEEN 6 PRECEDING AND CURRENT ROW, but some days are missing. What's wrong?

2. Days 10, 11, 12, 20 with row numbers 1–4. Values of day − row_number?

3. T-SQL equivalent of DuckDB's QUALIFY row_number() … = 1?

My notes