How netflix/metaflow works
Purpose and entry points
Metaflow is a Python-first framework for authoring, executing, tracking, and deploying AI/ML workflows (“flows”) from local development through cloud-scale production. The public runtime is the metaflow package; command invocation is centered on metaflow/cli.py and metaflow/cmd/main_cli.py. Users define flows by subclassing FlowSpec and marking step methods, primarily through metaflow/flowspec.py, metaflow/decorators.py, and metaflow/parameters.py.
Flow definition, graph inference, and execution
A flow’s step methods and self.next() transitions are converted into a DAG, including branches, foreach, parallelism, and switches. metaflow/graph.py models/inferes graph structure; metaflow/runtime.py schedules local task execution; metaflow/task.py executes an individual step/task. This is the core control boundary: user flow code is interpreted as a graph, then each graph node becomes an isolated task execution.
CLI and programmatic orchestration
The CLI exposes flow lifecycle commands such as run, resume, dump, and step execution via metaflow/cli_components/run_cmds.py, metaflow/cli_components/step_cmd.py, and metaflow/cli_components/dump_cmd.py. The Python Runner API launches and manages flows as subprocesses through metaflow/runner/metaflow_runner.py and metaflow/runner/subprocess_manager.py. These layers translate user commands/API calls into runtime task invocations and backend-specific operations.
Artifacts, metadata, and client access
Tasks persist artifacts, code/package references, logs, and lineage separately from execution control. The datastore abstraction and task-level persistence reside in metaflow/datastore/flow_datastore.py, metaflow/datastore/task_datastore.py, and metaflow/datastore/content_addressed_store.py; artifact serialization is handled under metaflow/datastore/artifacts/serializer.py. Metadata is supplied through configurable providers in metaflow/metadata_provider/metadata.py and metaflow/plugins/metadata_providers/service.py. The read-side client API navigates flows, runs, steps, tasks, and artifacts through metaflow/client/core.py.
Extensible execution and deployment backends
Decorators annotate flows/steps with execution requirements and behavior; plugin discovery makes backends replaceable or additive. Core extension mechanisms are in metaflow/decorators.py, metaflow/extension_support/plugins.py, and metaflow/plugins/__init__.py. Material compute/deployment targets include AWS Batch (metaflow/plugins/aws/batch/batch.py), Kubernetes (metaflow/plugins/kubernetes/kubernetes.py), Argo Workflows (metaflow/plugins/argo/argo_workflows.py), Airflow (metaflow/plugins/airflow/airflow.py), and AWS Step Functions (metaflow/plugins/aws/step_functions/step_functions.py).
Cloud services, environments, and observability
Datastore adapters support local files, Amazon S3, Azure Blob Storage, and Google Cloud Storage via metaflow/plugins/datastores/local_storage.py, metaflow/plugins/datastores/s3_storage.py, and metaflow/plugins/datastores/azure_storage.py. Dependency environments are provisioned by PyPI/Conda and uv plugins in metaflow/plugins/pypi/pypi_environment.py and metaflow/plugins/uv/uv_environment.py. Runtime logging, monitoring, tracing, and sidecars are separated from user tasks in metaflow/mflog/mflog.py, metaflow/tracing/propagator.py, and metaflow/sidecar/sidecar.py.
Presentation and language bindings
Cards render task/run results into portable visual artifacts and can be served/viewed through metaflow/plugins/cards/card_creator.py, metaflow/plugins/cards/card_server.py, and the Svelte UI at metaflow/plugins/cards/ui/src/App.svelte. The repository also ships an R package binding that constructs and controls Metaflow flows through R/R/flow.R, R/R/step.R, and R/inst/run.R, bridging R-authored workflows to the Python runtime.
Main entry points: task.py Task Execution · task_datastore.py Task Datastore · plugins.py Plugin Discovery · flowspec.py FlowSpec
Components 21
Flow Authoring
- FlowSpec · Users define steps and transitions in flow subclasses.
metaflow/flowspec.py - Decorators
metaflow/decorators.py - Parameters
metaflow/parameters.py - R Binding · Builds and controls Python-runtime flows from R.
R/R/flow.R
Control Plane
- Flow Graph · Infers step DAGs, branches, foreach, and switches.
metaflow/graph.py - CLI
metaflow/cli.py - Lifecycle Commands · Run, resume, dump, and step commands.
metaflow/cli_components/run_cmds.py - Runner API · Launches flows as managed subprocesses.
metaflow/runner/metaflow_runner.py - Local Runtime · Schedules graph nodes as isolated task executions.
metaflow/runtime.py - Task Execution
metaflow/task.py
Persistence & Access
- Task Datastore · Persists task data, code references, and logs.
metaflow/datastore/task_datastore.py - Artifact Serializer
metaflow/datastore/artifacts/serializer.py - Metadata Providers · Configurable run and lineage metadata providers.
metaflow/metadata_provider/metadata.py - Client API · Navigates flows, runs, steps, tasks, and artifacts.
metaflow/client/core.py
Extensions & Targets
- Plugin Discovery
metaflow/extension_support/plugins.py - Compute Backends · Batch and Kubernetes execute remote tasks.
metaflow/plugins/aws/batch/batch.py - Deployment Backends · Argo, Airflow, and Step Functions deployment targets.
metaflow/plugins/argo/argo_workflows.py - Cloud Datastores · Local, S3, Azure Blob, and GCS adapters.
metaflow/plugins/datastores/s3_storage.py - Environments · PyPI, Conda, and uv runtime environments.
metaflow/plugins/pypi/pypi_environment.py
Observability & Presentation
- Logging & Tracing · Logging, tracing, and sidecars are separated from user tasks.
metaflow/mflog/mflog.py - Cards · Renders task and run results as portable visual artifacts.
metaflow/plugins/cards/card_creator.py
Connections 22
- FlowSpec → Decorators: annotated by
- FlowSpec → Parameters: declares
- FlowSpec → Flow Graph: is inferred into
- R Binding → FlowSpec: constructs
- CLI → Lifecycle Commands: exposes
- Lifecycle Commands → Local Runtime: invokes
- Runner API → Local Runtime: launches
- Flow Graph → Local Runtime: schedules
- Local Runtime → Task Execution: executes nodes as
- Task Execution → Task Datastore: persists to
- Task Execution → Metadata Providers: records lineage
- Task Datastore → Artifact Serializer: serializes artifacts with
- Task Datastore → Cloud Datastores: uses adapters
- Client API → Task Datastore: reads artifacts
- Client API → Metadata Providers: navigates
- Decorators → Plugin Discovery: extends through
- Plugin Discovery → Compute Backends: discovers
- Plugin Discovery → Deployment Backends: discovers
- Plugin Discovery → Environments: discovers
- Compute Backends → Task Execution: runs remotely
- Task Execution → Logging & Tracing: emits logs and traces
- Task Datastore → Cards: supplies results