Week 7 · Run it in production › Day 45

Day 45. Choosing tools with a written rationale

1. Why this matters

Job ads list different tools: Airflow here, ADF there, Prefect somewhere else. You can't master all of them in 7 weeks, but you can understand the shared concepts (Weeks 1–6) and how each tool expresses them. That's what lets a senior person switch tools in a week.

2. Orchestrators

Airflow skills are still the most requested in ads; Dagster is increasingly common in modern stacks; ADF is everywhere in Microsoft enterprises.

The same pipeline in Airflow (TaskFlow API)

from airflow.sdk import dag, task          # Airflow 3; in Airflow 2: from airflow.decorators import dag, task
import pendulum

@dag(schedule="15 6 * * *", start_date=pendulum.datetime(2026, 9, 1, tz="America/Toronto"), catchup=True)
def hotel_daily():
    @task(retries=3)
    def land(ds=None):           # ds = logical date as YYYY-MM-DD, the "business date" for this run
        return pms.land_day(ds)
    @task
    def stage(path: str):
        return pms.stage(path)
    stage(land())
hotel_daily()

And in Prefect

from prefect import flow, task
@task(retries=3, retry_delay_seconds=30)
def land(bdate: str) -> str: return pms.land_day(bdate)
@flow
def hotel_daily(bdate: str): pms.stage(land(bdate))

Notice your pms core module didn't change. That's the payoff of the Day 15 rule.

3. Ingestion tools

• dlt: Python library, code-first, runs inside any orchestrator, cheap. Great for APIs and files.
• Airbyte: hundreds of connectors with a UI; self-host (needs a bit of infrastructure) or cloud.
• Meltano: Singer taps/targets with CLI and config, OSS.
• Fivetran / Azure Data Factory copy / Fabric: managed connectors, priced per volume/activity; the fastest start, the highest bill at scale.

4. Transformation

dbt (Core is open source; dbt Cloud adds hosted IDE/scheduling) is the default. SQLMesh is the strongest alternative, with virtual environments and plan/apply. Stored procedures remain common in SQL Server shops.