Orchestration: Airflow, Composer and Dagster

Scheduling concepts that trip people up, the task-centric vs asset-centric difference between Airflow and Dagster, and how to answer questions about retries, backfills and SLAs.

Orchestration is what turns a pile of scripts into a platform. Interviewers probe it because it reveals whether you have actually operated pipelines at 3am, or only written them.

Concepts that trip people up

Concept	What it really means
Logical date / data interval	The period the run covers, not when it executes. A daily DAG for 2026-03-01 typically runs at the end of 2026-03-01
Catchup	When enabled, a newly deployed DAG runs for every missed past interval. Leaving this on by accident is a classic incident
Backfill	Deliberately running past intervals to populate or correct history
Sensor	A task that waits for an external condition. Use `reschedule` mode, not `poke`, or it holds a worker slot for hours
Idempotency	Re-running an interval produces the same result. Non-negotiable, because retries are automatic
SLA / timeout	Alert when a task is late or hung, rather than silently waiting forever

Watch out — Derive the partition from the interval, never from now()

If a task computes current_date() internally, a rerun of a past date writes to today's partition and corrupts data. Always pass the logical date in and let the task write the partition it was asked for. This single habit is what makes backfills safe.

# Airflow: pass the interval in explicitly, so reruns and backfills are correct
run_kpi = BigQueryInsertJobOperator(
    task_id="build_daily_kpi",
    configuration={
        "query": {
            "query": "{{ var.value.kpi_sql }}",
            "useLegacySql": False,
            "queryParameters": [{
                "name": "run_date",
                "parameterType": {"type": "DATE"},
                # the DAG's data interval — NOT current_date()
                "parameterValue": {"value": "{{ ds }}"},
            }],
        }
    },
    retries=3,
    retry_delay=timedelta(minutes=5),
    retry_exponential_backoff=True,
    execution_timeout=timedelta(hours=2),
    sla=timedelta(hours=4),
)

Airflow vs Dagster

	Airflow / Cloud Composer	Dagster
Mental model	Task-centric — 'run these steps in this order'	Asset-centric — 'these tables should exist and be fresh'
Lineage	Task dependencies; data lineage is external	Data assets and their dependencies are first-class
Testing	Harder — logic often lives inside operators	Designed for local testing and typed I/O
Ecosystem	Huge; the industry default; managed as Composer/MWAA	Smaller but growing fast; strong dbt integration
Best for	Broad scheduling across many heterogeneous systems	Data-platform work where assets and freshness are the unit of thought

Tip — The nuanced answer if you have used both

'Airflow is excellent at scheduling arbitrary operational tasks across many systems. Dagster fits better where the unit of work is a data asset — you declare that a table should be fresh, get lineage and partition awareness for free, and testing is much easier. Running both is common: Composer for platform-wide scheduling, Dagster for the transformation graph.'

Practice questions

QA task fails intermittently at 2am. How do you make the pipeline resilient?show

Distinguish transient failures (retry them) from deterministic ones (retrying will never help), then make the retry safe.

Retries with exponential backoff for transient network/API failures
Idempotent tasks so a retry cannot duplicate data — partition overwrite, not append
Timeouts so a hung task fails fast instead of blocking the schedule
Sensors in reschedule mode so waiting does not consume a worker slot
Alert on repeated retries, not just on final failure — a task that always needs 3 attempts is a warning sign
Runbook attached to the alert so whoever is on call knows the safe recovery action

QHow do you handle dependencies between DAGs owned by different teams?show

Avoid tight cross-team coupling; depend on data being ready rather than on someone else's task finishing.

Data-aware scheduling — trigger on a dataset being updated (Airflow datasets, Dagster asset sensors)
Sensor on the data, not on the upstream DAG's task state — the upstream team can then refactor freely
Avoid deep TriggerDagRunOperator chains; they create brittle, hard-to-debug coupling
Back it with a data contract: freshness SLA plus quality guarantees, so the dependency is explicit and testable

QYour DAG has 500 tasks and the scheduler is struggling. What do you do?show

Reduce task count and parallelism pressure, and check whether the DAG is doing work it should delegate.

Dynamic task mapping instead of generating hundreds of near-identical static tasks
Group work — one task that processes a batch, rather than one task per file
Push compute out — the DAG should trigger BigQuery/Spark jobs, not process data inside the worker
Tune pools and concurrency so one DAG cannot starve the whole environment
Split by domain into several DAGs with data-aware dependencies between them

← PreviousData Quality, Reliability and Observability Next →CI/CD, Docker and Kubernetes for Data Engineers