A daily data pipeline that fixes itself, run by a crew of agents with no boss
Retail, hospitality, restaurant and logistics chains all run the same nightly routine: collect the day's exports, clean them, load a warehouse and refresh the morning reports. This page puts the usual scheduled pipeline beside the same pipeline run by a decentralised crew of AI agents, step by step, with the maths and a simulation to show what changes.
Nihal Vellaramkallingal Sulaiman · design and simulation, November 2026
Median time until the data is right again
per incident, simulated
People hours spent on breakages a year
crew figure includes 2 h a week of upkeep
Wrong numbers that reach a report unnoticed
silent errors a year
Model cost of the crew
a year, at current list prices
Scheduled pipeline one scheduler, one person on call
Decentralised agent crew same jobs, nine agents, one shared board
00
Who is in charge
Centralised
One scheduler decides what runs, and one person decides what to do when it fails. Everything waits on them, one problem at a time. When that person is busy or away, nothing moves.
Decentralised
There is no boss agent. Each agent has one duty and one rule for when to act, and they all read and write the same board. Nobody hands work to anybody: an agent sees an incident in the state it handles, claims it, does its part and posts the result. Like a small team in sync, the whole crew moves the system toward one shared goal, every asset fresh by 07:00.
Four rules make it behave like one team.
Act on state, not orders. "Evidence posted, no verdict yet" is the Triager's cue. "Verdict is real fault, no patch yet" is the Fixer's.
Claim with a lease. A claim expires after a few minutes, so if an agent stalls, a spare copy picks the task up.
One writer per field. Only the Triager writes verdicts and only the Verifier writes test results, so agents never overwrite each other.
Append, never edit. The board and the Ledger only grow, so any step can be traced and reverted.
01
The nightly run, start to finish
A schedule fires at 05:00 and runs each job in order. When one fails, everything after it waits. Nobody looks until working hours, and then a person has to find the cause, fix it and re-run by hand.
The same jobs run on the same schedule. A sensor notices the failure within a minute, a crew of small agents works out why, tests a fix in a copy of the pipeline, and either applies it by itself (safe re-runs) or asks a person with one click. Every step is written to a ledger that can undo it.
02
Collect the dirty data
How it is usually done
Daily exports arrive as email attachments, files in a shared folder, or an SFTP drop: sales, stock, deliveries, labour hours, online orders.
A rule or a person saves each one into a landing folder. Some exports are run by hand each morning.
The job reads whatever file is there at 05:00.
Where it breaks. A file is late, missing, sent twice, half-empty, or arrives with a renamed column. The job either fails or, worse, loads it and carries on.
Same step, with the crew
Land, never overwrite. Each file goes to landing/<source>/<business_date>/ with its SHA-256 hash, size, sender and arrival time in a manifest table.
Keep the original. A read-only copy goes to raw/. Nothing ever edits it.
Expect the file. Each source has a contract: expected arrival window, columns and types, and a normal row-count range.
Sense, don't poll. A Dagster sensor starts the load when the file lands, not when the clock says so.
What the crew adds. A late file is re-run by itself the moment it lands. A duplicate is spotted by its hash. A missing manual export becomes a drafted note to its owner, sent by a person with one click.
03
Clean it
How it is usually done
Cleaning code assumes yesterday's layout: fixed column names, date formats and units.
Bad rows are dropped or fixed silently inside the job.
Checks run after the load, if at all.
Where it breaks. A renamed column fails the job. A changed unit or a double-counted refund passes every check and lands in the report.
Same step, with the crew
Check the contract first: header, types, nulls in key columns, duplicates on the business key, row count against the last 4 weeks and the same weekday last year.
Quarantine, don't drop. Rows that fail go to a quarantine table with the rule they broke, so they can be counted and fixed later.
Standardise: one date format, one currency and unit, trimmed text, mapped codes (store, item, channel) through lookup tables, not code.
Tie out: totals in staging must match totals in the source file before anything moves on.
What the crew adds. A failed contract opens an incident with the evidence attached: yesterday's header next to today's, the rows that failed and why. The Triager reads that small file, not the whole log.
04
Save it to the data warehouse
How it is usually done
Append the day's rows to the fact tables.
Re-running a job means deleting rows by hand first, or risking doubles.
Where it breaks. A re-run after a partial load doubles a day's sales. Fixing it needs someone who knows which rows to delete.
Same step, with the crew
Partition by business date. Every asset is a Dagster asset partitioned by day.
Make every load idempotent:MERGE on the business key, or replace the whole partition. Running it twice gives the same result as once.
Star schema: facts (sales, stock, deliveries, hours) at a stated grain, with conformed dimensions (date, location, product, channel).
Asset checks after each load: keys unique, totals tie out, nothing sold above capacity, no gaps in dates.
What the crew adds. Because loads are idempotent, the crew may re-run a partition by itself. This one design choice is what makes most incidents safe to fix without a person.
05
Reports and dashboards, live every morning
How it is usually done
Dashboards refresh on their own timer, whether or not the data under them loaded.
A refreshed dashboard looks the same whether its data is from today or yesterday.
Where it breaks. Managers read yesterday's numbers as today's. The first sign of trouble is often a question from a reader.
Same step, with the crew
Refresh on success, not on a timer: the report refresh is the last asset in the graph, so it runs only when its inputs are fresh.
Show freshness on the page: each dashboard carries a "data as of" stamp and a banner when a source is late.
Refresh again after a fix: when the crew repairs a source at 05:33, the reports refresh by 07:00 as if nothing happened.
One morning note: the Reporter writes a short note listing what broke overnight, what was fixed, and what still needs a person.
What the crew adds. The command centre shows each agent's work as it happens, with buttons to pause, stop, approve, revert a step, or rework from it.
06
When something breaks: six common faults
Handled by a person
Fault
What happens
Median hours
Handled by the crew
Fault
What happens
Median hours
The maths of working without a boss
Two properties come from spreading the work across independent agents instead of one central point. Errors have to slip past several different checks, and no single failure can stop the whole crew.
Independent eyes. If m agents each catch a bad file with chance di, independently:
P(missed by all) = (1 − d1)(1 − d2)…(1 − dm)
One rule at 0.75: slip through
Four agents at 0.6, 0.5, 0.5, 0.5: slip through
No single point of failure. If one instance is down on a night with chance f:
Central coordinator: P(everything stalls) = f =
Two copies of each role: P(one role stalls) = f2 =
Any of 9 roles stalls: 1 − (1 − f2)9 =
Chance a bad file reaches a report
Each extra independent check (each catching half of bad files) halves what slips through. This holds only while the checks look at different things.
Why breakage is the normal case
Each source has a small chance of a problem on any day. Small chances multiply. With daily sources and about chance of a problem per source per day, fewer than half of all nights run clean.
P(clean night) = (1 − p)n × (1 − b)
= (1 − ) × (1 − ) =
Expected incidents a night: λ = n·p + b =
Incidents a year ≈ 365 λ =
Chance that a night runs clean
Chance of a clean night as the number of daily sources grows, at per source. The marked point is this pipeline.
How long reports stay wrong
Simulated over years of nightly runs with the same faults on both sides. The crew fixes half of all incidents within about an hour. The long tail on both sides is waiting for people outside the team, such as an export owner re-sending a file.
Scheduled pipelineAgent crew
Share of incidents where the data is right again within a given time (log scale). Hover to read values.
Yearly totals, simulated. Report-hours late adds up how many hours each affected report missed its 07:00 deadline.
Why a few retries are enough
If the Fixer's patch passes the tests in the dev copy with chance q on each try, and each try learns from the last failure, the chance of a fix within k tries is:
P(fixed within k tries) = 1 − (1 − q)k
with q = : 1 try · 2 tries · 3 tries
Cost of a worst case: 2 × Sonnet + 1 × Opus ≈ $0.19
Expected model cost a year ≈
After three tries the crew stops and hands the full trail to a person. Past that point a fourth try adds little and costs more.
Limits, as of now
The numbers are a model
Fault rates, times and costs are stated assumptions, not measurements. Replace them with a real pipeline's run history to get real figures.
People outside the team are still slow
When a manual export is missing, the crew can only draft the note faster. The wait for the sender stays.
Code changes still need a person
The crew tests every patch, but merging into production is a one-click human decision by design.
Idempotent loads come first
Safe automatic re-runs depend on loads that can run twice. A pipeline without that must be fixed before the crew is switched on.
Models can be wrong
A wrong verdict is caught by the Verifier's tests, not by trust. Checks that are missing cannot catch anything.
Secrets and access
The crew reads production through a read-only login and writes only to a dev copy and a git branch.
Connect it to your own systems
This page and the command centre run on a simulation: no live database or pipeline sits behind them. To run the crew for real, connect these points in this order. The setup guide walks through each step.
Part
In this demo
Connect yours here
Setting
Daily exports
Simulated faults
Shared mailbox rule or folder, SFTP drop or API pull
LANDING_ROOT
Source contracts
Built into the script
One YAML file per source
contracts/<source>.yaml
Orchestrator
Scripted timeline
Dagster, with a sensor per source
DAGSTER_GRAPHQL_URL
Warehouse and dev copy
None
Azure SQL, SQL Server, PostgreSQL, Snowflake or similar
WAREHOUSE_URL, WAREHOUSE_DEV_URL
Read-only access for the crew
None
A login that can only read
CREW_RO_URL
Shared board and ledger
In the page's memory
Two tables in an ops schema and a git repository
BOARD_URL, PIPELINE_REPO
Language models
Fixed token counts
A Claude API key with a monthly spend limit
ANTHROPIC_API_KEY
Dashboards
Asset list only
Power BI or Tableau refresh API
POWERBI_DATASET_ID or TABLEAU_EXTRACT_ID
Command centre
Scripted night
A small API over the board tables
/api/board
Assumptions and how to reproduce
Assumption
Value
The simulation is one Python file with no dependencies. Change any value and run python simulate.py. Download simulate.py and its results.json.
Figures come from a simulation with stated assumptions, not from any company's data.