Pipeline Orchestration

A trained model isn't produced by one script — it's produced by a sequence of steps that each depend on the last, need to run on a schedule, and need to recover cleanly when one of them fails at 3am. That's a pipeline orchestration problem, not a training problem.

The ML Lifecycle as a Pipeline

Validation → Preprocessing → Feature Engineering → Training → Evaluation → Registry → Deployment → Monitoring → Retraining

Each stage takes the previous stage's output as input. Running this by hand (or as one giant script) works exactly once — a real system needs to run this on a schedule, retry the step that failed without re-running everything before it, and show a human which step broke.

Core Concepts

• DAG (Directed Acyclic Graph): the pipeline as a graph of steps with dependencies — step C can't start until steps A and B finish, and there are no cycles. This is the data structure every orchestrator is built around.

• Task / Operator: a single unit of work in the DAG (e.g. "run this Python function," "run this SQL query," "run this Docker container").
• Scheduling: run the DAG on a cron-like schedule (nightly retraining) or trigger it on an event (new data landed).
• Retries: if a task fails (a flaky API call, a transient network error), retry it automatically before escalating to a human.

• Caching: skip re-running a step whose inputs haven't changed since the last successful run — the same principle as a build system, applied to a data pipeline.

• Backfilling: run the pipeline for a range of past dates, e.g. after fixing a bug in a transformation step, without hand-rerunning each day individually.

Airflow

The most widely used orchestrator, and the one whose vocabulary shows up everywhere else:

• DAG: a Python file defining the pipeline structure.
• Task: one node in the DAG.
• Operator: the type of work a task does (PythonOperator, BashOperator, KubernetesPodOperator, etc.).
• Scheduler: the process that decides when each DAG run should trigger.
• Executor: the process that actually runs tasks (locally, on Celery workers, or on Kubernetes pods, depending on configuration).
• Sensor: a special task type that waits for a condition (a file landing in S3, a partition appearing in a table) before letting downstream tasks proceed.
• XCom: Airflow's mechanism for passing small pieces of data between tasks in the same DAG run.

Airflow's model — DAGs as Python code, a scheduler, pluggable executors — is old enough (originally built at Airbnb) to be the de facto lingua franca; most job postings asking for "pipeline orchestration experience" mean Airflow specifically.

A real DAG covering the training slice of the lifecycle above (validation → preprocessing → training → evaluation), using Airflow's modern TaskFlow API:

from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="@nightly", start_date=datetime(2026, 1, 1), catchup=False)
def retrain_fraud_model():
    @task(retries=3, retry_delay=300)
    def validate_data():
        return check_schema_and_ranges("s3://data/fraud/latest")

    @task
    def preprocess(validated_path: str):
        return build_features(validated_path)

    @task
    def train(feature_path: str):
        return run_training(feature_path)  # logs to MLflow, see Experiment Tracking

    @task
    def evaluate(run_id: str):
        if not beats_production_benchmark(run_id):
            raise ValueError("candidate did not beat production -- stopping before registry/deploy")

    evaluate(train(preprocess(validate_data())))

retrain_fraud_model()

airflow dags trigger retrain_fraud_model              # run it now, outside the schedule
airflow tasks test retrain_fraud_model validate_data 2026-01-15   # run one task in isolation, for debugging
airflow dags backfill retrain_fraud_model -s 2026-01-01 -e 2026-01-07   # rerun a past date range

Alternatives

• Prefect: a newer orchestrator with a lower-friction Python-native API (decorate a normal function with @flow/@task rather than building an explicit DAG object), stronger dynamic/conditional workflow support, and a more modern UI.

from prefect import flow, task

@task(retries=3)
def validate_data():
    return check_schema_and_ranges("s3://data/fraud/latest")

@task
def train(feature_path: str):
    return run_training(feature_path)

@flow
def retrain_fraud_model():
    validated = validate_data()
    train(validated)   # same dependency graph, no explicit DAG object -- inferred from the call order

• Dagster: orchestration built around software-defined assets rather than tasks — you declare the data assets a pipeline produces and their dependencies, and Dagster infers the DAG, with strong built-in data quality and lineage tracking.

Airflow remains the safest default to learn first because of its ubiquity; Prefect and Dagster are the ones to reach for on a new project where Airflow's more verbose, less Pythonic API is a real cost.

Next: Message Queues & Async Processing — the mechanism underneath Celery workers (and streaming pipelines generally) that this page only referenced in passing.