# netflix/metaflow architecture

> Architecture diagram of the GitHub repository netflix/metaflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netflix/metaflow
- Repository: https://github.com/netflix/metaflow

## Overview

#### Purpose and entry points
Metaflow is a Python-first framework for authoring, executing, tracking, and deploying AI/ML workflows (“flows”) from local development through cloud-scale production. The public runtime is the `metaflow` package; command invocation is centered on `metaflow/cli.py` and `metaflow/cmd/main_cli.py`. Users define flows by subclassing `FlowSpec` and marking step methods, primarily through `metaflow/flowspec.py`, `metaflow/decorators.py`, and `metaflow/parameters.py`.

#### Flow definition, graph inference, and execution
A flow’s step methods and `self.next()` transitions are converted into a DAG, including branches, `foreach`, parallelism, and switches. `metaflow/graph.py` models/inferes graph structure; `metaflow/runtime.py` schedules local task execution; `metaflow/task.py` executes an individual step/task. This is the core control boundary: user flow code is interpreted as a graph, then each graph node becomes an isolated task execution.

#### CLI and programmatic orchestration
The CLI exposes flow lifecycle commands such as run, resume, dump, and step execution via `metaflow/cli_components/run_cmds.py`, `metaflow/cli_components/step_cmd.py`, and `metaflow/cli_components/dump_cmd.py`. The Python Runner API launches and manages flows as subprocesses through `metaflow/runner/metaflow_runner.py` and `metaflow/runner/subprocess_manager.py`. These layers translate user commands/API calls into runtime task invocations and backend-specific operations.

#### Artifacts, metadata, and client access
Tasks persist artifacts, code/package references, logs, and lineage separately from execution control. The datastore abstraction and task-level persistence reside in `metaflow/datastore/flow_datastore.py`, `metaflow/datastore/task_datastore.py`, and `metaflow/datastore/content_addressed_store.py`; artifact serialization is handled under `metaflow/datastore/artifacts/serializer.py`. Metadata is supplied through configurable providers in `metaflow/metadata_provider/metadata.py` and `metaflow/plugins/metadata_providers/service.py`. The read-side client API navigates flows, runs, steps, tasks, and artifacts through `metaflow/client/core.py`.

#### Extensible execution and deployment backends
Decorators annotate flows/steps with execution requirements and behavior; plugin discovery makes backends replaceable or additive. Core extension mechanisms are in `metaflow/decorators.py`, `metaflow/extension_support/plugins.py`, and `metaflow/plugins/__init__.py`. Material compute/deployment targets include AWS Batch (`metaflow/plugins/aws/batch/batch.py`), Kubernetes (`metaflow/plugins/kubernetes/kubernetes.py`), Argo Workflows (`metaflow/plugins/argo/argo_workflows.py`), Airflow (`metaflow/plugins/airflow/airflow.py`), and AWS Step Functions (`metaflow/plugins/aws/step_functions/step_functions.py`).

#### Cloud services, environments, and observability
Datastore adapters support local files, Amazon S3, Azure Blob Storage, and Google Cloud Storage via `metaflow/plugins/datastores/local_storage.py`, `metaflow/plugins/datastores/s3_storage.py`, and `metaflow/plugins/datastores/azure_storage.py`. Dependency environments are provisioned by PyPI/Conda and uv plugins in `metaflow/plugins/pypi/pypi_environment.py` and `metaflow/plugins/uv/uv_environment.py`. Runtime logging, monitoring, tracing, and sidecars are separated from user tasks in `metaflow/mflog/mflog.py`, `metaflow/tracing/propagator.py`, and `metaflow/sidecar/sidecar.py`.

#### Presentation and language bindings
Cards render task/run results into portable visual artifacts and can be served/viewed through `metaflow/plugins/cards/card_creator.py`, `metaflow/plugins/cards/card_server.py`, and the Svelte UI at `metaflow/plugins/cards/ui/src/App.svelte`. The repository also ships an R package binding that constructs and controls Metaflow flows through `R/R/flow.R`, `R/R/step.R`, and `R/inst/run.R`, bridging R-authored workflows to the Python runtime.

Main entry points: [`metaflow/task.py`](https://github.com/netflix/metaflow/blob/master/metaflow/task.py) (Task Execution), [`metaflow/datastore/task_datastore.py`](https://github.com/netflix/metaflow/blob/master/metaflow/datastore/task_datastore.py) (Task Datastore), [`metaflow/extension_support/plugins.py`](https://github.com/netflix/metaflow/blob/master/metaflow/extension_support/plugins.py) (Plugin Discovery), [`metaflow/flowspec.py`](https://github.com/netflix/metaflow/blob/master/metaflow/flowspec.py) (FlowSpec)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_authoring["Flow Authoring"]
  node_flow_spec["FlowSpec<br/>flow authoring<br/>[flowspec.py]"]
  node_decorators["Decorators<br/>flow annotations<br/>[decorators.py]"]
  node_parameters["Parameters<br/>flow inputs<br/>[parameters.py]"]
  node_r_binding["R Binding<br/>language binding<br/>[flow.R]"]
end

subgraph group_control["Control Plane"]
  node_graph["Flow Graph<br/>DAG inference<br/>[graph.py]"]
  node_cli["CLI<br/>command entrypoint<br/>[cli.py]"]
  node_cli_commands["Lifecycle Commands<br/>CLI commands<br/>[run_cmds.py]"]
  node_runner["Runner API<br/>Python orchestration<br/>[metaflow_runner.py]"]
  node_runtime["Local Runtime<br/>scheduler<br/>[runtime.py]"]
  node_task["Task Execution<br/>task runtime<br/>[task.py]"]
end

subgraph group_persistence["Persistence &amp; Access"]
  node_datastore[("Task Datastore<br/>persistence abstraction<br/>[task_datastore.py]")]
  node_artifact_serializer["Artifact Serializer<br/>serialization<br/>[serializer.py]"]
  node_metadata[("Metadata Providers<br/>lineage metadata<br/>[metadata.py]")]
  node_client["Client API<br/>read API<br/>[core.py]"]
end

subgraph group_extensions["Extensions &amp; Targets"]
  node_plugins["Plugin Discovery<br/>extension registry<br/>[plugins.py]"]
  node_compute_targets{{"Compute Backends<br/>execution targets<br/>[batch.py]"}}
  node_deployment_targets{{"Deployment Backends<br/>workflow deployment<br/>[argo_workflows.py]"}}
  node_cloud_datastores[("Cloud Datastores<br/>storage adapters<br/>[s3_storage.py]")]
  node_environments["Environments<br/>dependency provisioning"]
end

subgraph group_presentation["Observability &amp; Presentation"]
  node_observability["Logging &amp; Tracing<br/>runtime observability<br/>[mflog.py]"]
  node_cards["Cards<br/>result presentation<br/>[card_creator.py]"]
end

node_flow_spec -->|"annotated by"| node_decorators
node_flow_spec -->|"declares"| node_parameters
node_flow_spec -->|"is inferred into"| node_graph
node_r_binding -->|"constructs"| node_flow_spec
node_cli -->|"exposes"| node_cli_commands
node_cli_commands -->|"invokes"| node_runtime
node_runner -->|"launches"| node_runtime
node_graph -->|"schedules"| node_runtime
node_runtime -->|"executes nodes as"| node_task
node_task -->|"persists to"| node_datastore
node_task -->|"records lineage"| node_metadata
node_datastore -->|"serializes artifacts with"| node_artifact_serializer
node_datastore -->|"uses adapters"| node_cloud_datastores
node_client -->|"reads artifacts"| node_datastore
node_client -->|"navigates"| node_metadata
node_decorators -->|"extends through"| node_plugins
node_plugins -->|"discovers"| node_compute_targets
node_plugins -->|"discovers"| node_deployment_targets
node_plugins -->|"discovers"| node_environments
node_compute_targets -.->|"runs remotely"| node_task
node_task -->|"emits logs and traces"| node_observability
node_datastore -->|"supplies results"| node_cards

click node_flow_spec "https://github.com/netflix/metaflow/blob/master/metaflow/flowspec.py"
click node_decorators "https://github.com/netflix/metaflow/blob/master/metaflow/decorators.py"
click node_parameters "https://github.com/netflix/metaflow/blob/master/metaflow/parameters.py"
click node_graph "https://github.com/netflix/metaflow/blob/master/metaflow/graph.py"
click node_cli "https://github.com/netflix/metaflow/blob/master/metaflow/cli.py"
click node_cli_commands "https://github.com/netflix/metaflow/blob/master/metaflow/cli_components/run_cmds.py"
click node_runner "https://github.com/netflix/metaflow/blob/master/metaflow/runner/metaflow_runner.py"
click node_runtime "https://github.com/netflix/metaflow/blob/master/metaflow/runtime.py"
click node_task "https://github.com/netflix/metaflow/blob/master/metaflow/task.py"
click node_datastore "https://github.com/netflix/metaflow/blob/master/metaflow/datastore/task_datastore.py"
click node_artifact_serializer "https://github.com/netflix/metaflow/blob/master/metaflow/datastore/artifacts/serializer.py"
click node_metadata "https://github.com/netflix/metaflow/blob/master/metaflow/metadata_provider/metadata.py"
click node_client "https://github.com/netflix/metaflow/blob/master/metaflow/client/core.py"
click node_plugins "https://github.com/netflix/metaflow/blob/master/metaflow/extension_support/plugins.py"
click node_compute_targets "https://github.com/netflix/metaflow/blob/master/metaflow/plugins/aws/batch/batch.py"
click node_deployment_targets "https://github.com/netflix/metaflow/blob/master/metaflow/plugins/argo/argo_workflows.py"
click node_cloud_datastores "https://github.com/netflix/metaflow/blob/master/metaflow/plugins/datastores/s3_storage.py"
click node_environments "https://github.com/netflix/metaflow/blob/master/metaflow/plugins/pypi/pypi_environment.py"
click node_observability "https://github.com/netflix/metaflow/blob/master/metaflow/mflog/mflog.py"
click node_cards "https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/card_creator.py"
click node_r_binding "https://github.com/netflix/metaflow/blob/master/R/R/flow.R"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_flow_spec,node_decorators,node_parameters,node_r_binding toneBlue
class node_graph,node_cli,node_cli_commands,node_runner,node_runtime,node_task toneAmber
class node_datastore,node_artifact_serializer,node_metadata,node_client toneMint
class node_plugins,node_compute_targets,node_deployment_targets,node_cloud_datastores,node_environments toneRose
class node_observability,node_cards toneIndigo
```

## Components

### Flow Authoring

- [FlowSpec](https://github.com/netflix/metaflow/blob/master/metaflow/flowspec.py): flow authoring · Users define steps and transitions in flow subclasses. · `metaflow/flowspec.py`
- [Decorators](https://github.com/netflix/metaflow/blob/master/metaflow/decorators.py): flow annotations · `metaflow/decorators.py`
- [Parameters](https://github.com/netflix/metaflow/blob/master/metaflow/parameters.py): flow inputs · `metaflow/parameters.py`
- [R Binding](https://github.com/netflix/metaflow/blob/master/R/R/flow.R): language binding · Builds and controls Python-runtime flows from R. · `R/R/flow.R`

### Control Plane

- [Flow Graph](https://github.com/netflix/metaflow/blob/master/metaflow/graph.py): DAG inference · Infers step DAGs, branches, foreach, and switches. · `metaflow/graph.py`
- [CLI](https://github.com/netflix/metaflow/blob/master/metaflow/cli.py): command entrypoint · `metaflow/cli.py`
- [Lifecycle Commands](https://github.com/netflix/metaflow/blob/master/metaflow/cli_components/run_cmds.py): CLI commands · Run, resume, dump, and step commands. · `metaflow/cli_components/run_cmds.py`
- [Runner API](https://github.com/netflix/metaflow/blob/master/metaflow/runner/metaflow_runner.py): Python orchestration · Launches flows as managed subprocesses. · `metaflow/runner/metaflow_runner.py`
- [Local Runtime](https://github.com/netflix/metaflow/blob/master/metaflow/runtime.py): scheduler · Schedules graph nodes as isolated task executions. · `metaflow/runtime.py`
- [Task Execution](https://github.com/netflix/metaflow/blob/master/metaflow/task.py): task runtime · `metaflow/task.py`

### Persistence & Access

- [Task Datastore](https://github.com/netflix/metaflow/blob/master/metaflow/datastore/task_datastore.py): persistence abstraction · Persists task data, code references, and logs. · `metaflow/datastore/task_datastore.py`
- [Artifact Serializer](https://github.com/netflix/metaflow/blob/master/metaflow/datastore/artifacts/serializer.py): serialization · `metaflow/datastore/artifacts/serializer.py`
- [Metadata Providers](https://github.com/netflix/metaflow/blob/master/metaflow/metadata_provider/metadata.py): lineage metadata · Configurable run and lineage metadata providers. · `metaflow/metadata_provider/metadata.py`
- [Client API](https://github.com/netflix/metaflow/blob/master/metaflow/client/core.py): read API · Navigates flows, runs, steps, tasks, and artifacts. · `metaflow/client/core.py`

### Extensions & Targets

- [Plugin Discovery](https://github.com/netflix/metaflow/blob/master/metaflow/extension_support/plugins.py): extension registry · `metaflow/extension_support/plugins.py`
- [Compute Backends](https://github.com/netflix/metaflow/blob/master/metaflow/plugins/aws/batch/batch.py): execution targets · Batch and Kubernetes execute remote tasks. · `metaflow/plugins/aws/batch/batch.py`
- [Deployment Backends](https://github.com/netflix/metaflow/blob/master/metaflow/plugins/argo/argo_workflows.py): workflow deployment · Argo, Airflow, and Step Functions deployment targets. · `metaflow/plugins/argo/argo_workflows.py`
- [Cloud Datastores](https://github.com/netflix/metaflow/blob/master/metaflow/plugins/datastores/s3_storage.py): storage adapters · Local, S3, Azure Blob, and GCS adapters. · `metaflow/plugins/datastores/s3_storage.py`
- [Environments](https://github.com/netflix/metaflow/blob/master/metaflow/plugins/pypi/pypi_environment.py): dependency provisioning · PyPI, Conda, and uv runtime environments. · `metaflow/plugins/pypi/pypi_environment.py`

### Observability & Presentation

- [Logging & Tracing](https://github.com/netflix/metaflow/blob/master/metaflow/mflog/mflog.py): runtime observability · Logging, tracing, and sidecars are separated from user tasks. · `metaflow/mflog/mflog.py`
- [Cards](https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/card_creator.py): result presentation · Renders task and run results as portable visual artifacts. · `metaflow/plugins/cards/card_creator.py`

## Connections

- FlowSpec → Decorators: annotated by
- FlowSpec → Parameters: declares
- FlowSpec → Flow Graph: is inferred into
- R Binding → FlowSpec: constructs
- CLI → Lifecycle Commands: exposes
- Lifecycle Commands → Local Runtime: invokes
- Runner API → Local Runtime: launches
- Flow Graph → Local Runtime: schedules
- Local Runtime → Task Execution: executes nodes as
- Task Execution → Task Datastore: persists to
- Task Execution → Metadata Providers: records lineage
- Task Datastore → Artifact Serializer: serializes artifacts with
- Task Datastore → Cloud Datastores: uses adapters
- Client API → Task Datastore: reads artifacts
- Client API → Metadata Providers: navigates
- Decorators → Plugin Discovery: extends through
- Plugin Discovery → Compute Backends: discovers
- Plugin Discovery → Deployment Backends: discovers
- Plugin Discovery → Environments: discovers
- Compute Backends → Task Execution: runs remotely
- Task Execution → Logging & Tracing: emits logs and traces
- Task Datastore → Cards: supplies results
