# dagster-io/dagster architecture

> Architecture diagram of the GitHub repository dagster-io/dagster, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dagster-io/dagster
- Repository: https://github.com/dagster-io/dagster

## Overview

Dagster is a cloud-native data asset and pipeline orchestrator. The principal workflow is to declare assets, have Dagster schedule and execute their computations, persist results, and inspect lineage and run health through its control-plane UI. The tree also exposes deployment configuration, BigQuery integrations, and UI support for runs, partitions, backfills, and cached queries. The supplied source excerpts are selective, so links to unsampled runtime subsystems are included only where the README documents the workflow; detailed internal wiring is not inferred.

Main entry points: [`js_modules/ui-core/src/app`](https://github.com/dagster-io/dagster/tree/master/js_modules/ui-core/src/app) (UI application core), [`python_modules/dagster/dagster`](https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster) (Asset definitions), [`python_modules/dagster/dagster/_core`](https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster/_core) (Scheduling and runs), [`js_modules/app-oss/src/App.tsx`](https://github.com/dagster-io/dagster/blob/master/js_modules/app-oss/src/App.tsx) (Dagster web app)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_definitions["Definitions and execution"]
  node_asset_defs["Asset definitions"]
  node_pipeline_graph["Asset lineage graph"]
  node_compute["Asset computation"]
end

subgraph group_platform["Orchestration platform"]
  node_orchestrator["Orchestration engine"]
  node_scheduling["Scheduling and runs"]
  node_deployment["Kubernetes deployment"]
end

subgraph group_control["Control plane UI"]
  node_web_app["Dagster web app<br/>[App.tsx]"]
  node_ui_core["UI application core"]
  node_asset_ui["Asset explorer"]
  node_run_ui["Run and partition views"]
  node_backfill_ui["Backfill details"]
  node_session_storage["Launchpad sessions"]
end

subgraph group_integrations["Storage and integrations"]
  node_cloud_logs["Compute log storage"]
  node_bigquery_pandas["BigQuery pandas I/O"]
  node_bigquery_spark["BigQuery Spark I/O"]
end

node_developer(("Data developer"))
node_bigquery[("Google BigQuery")]
node_browser_state[("Browser state cache")]

node_developer -->|"declares assets"| node_asset_defs
node_asset_defs -->|"defines dependencies"| node_pipeline_graph
node_asset_defs -->|"submits definitions"| node_orchestrator
node_orchestrator -->|"plans and launches"| node_scheduling
node_scheduling -->|"executes"| node_compute
node_compute -.->|"uses I/O manager"| node_bigquery_pandas
node_compute -.->|"uses I/O manager"| node_bigquery_spark
node_bigquery_pandas -->|"reads and writes"| node_bigquery
node_bigquery_spark -->|"reads and writes"| node_bigquery
node_scheduling -->|"captures logs"| node_cloud_logs
node_developer -->|"opens control plane"| node_web_app
node_web_app -->|"renders application"| node_ui_core
node_ui_core -->|"shows assets"| node_asset_ui
node_ui_core -->|"queries partition runs"| node_run_ui
node_ui_core -->|"queries backfills"| node_backfill_ui
node_ui_core -->|"persists launchpad state"| node_session_storage
node_session_storage -->|"reads and writes"| node_browser_state
node_asset_ui -->|"visualizes lineage"| node_pipeline_graph
node_deployment -.->|"configures deployment"| node_orchestrator

click node_asset_defs "https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster"
click node_pipeline_graph "https://github.com/dagster-io/dagster/tree/master/js_modules/ui-core/src/asset-graph"
click node_compute "https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster"
click node_orchestrator "https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster/_core"
click node_scheduling "https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster/_core"
click node_deployment "https://github.com/dagster-io/dagster/tree/master/helm/dagster/schema/schema/charts/dagster"
click node_web_app "https://github.com/dagster-io/dagster/blob/master/js_modules/app-oss/src/App.tsx"
click node_ui_core "https://github.com/dagster-io/dagster/tree/master/js_modules/ui-core/src/app"
click node_asset_ui "https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/asset-graph/AssetGraphExplorer.tsx"
click node_run_ui "https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/partitions/usePartitionStepQuery.tsx"
click node_backfill_ui "https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/instance/backfill/useBackfillDetailsQuery.tsx"
click node_session_storage "https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/app/ExecutionSessionStorage.tsx"
click node_cloud_logs "https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/cloud_storage_compute_log_manager.py"
click node_bigquery_pandas "https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pandas/dagster_gcp_pandas/bigquery/bigquery_pandas_type_handler.py"
click node_bigquery_spark "https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pyspark/dagster_gcp_pyspark/bigquery/bigquery_pyspark_type_handler.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_asset_defs,node_pipeline_graph,node_compute toneBlue
class node_orchestrator,node_scheduling,node_deployment,node_bigquery,node_browser_state toneAmber
class node_web_app,node_ui_core,node_asset_ui,node_run_ui,node_backfill_ui,node_session_storage toneMint
class node_cloud_logs,node_bigquery_pandas,node_bigquery_spark toneRose
class node_developer toneIndigo
```

## Components

### Definitions and execution

- [Asset definitions](https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster): `python_modules/dagster/dagster`
- [Asset lineage graph](https://github.com/dagster-io/dagster/tree/master/js_modules/ui-core/src/asset-graph): `js_modules/ui-core/src/asset-graph`
- [Asset computation](https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster): `python_modules/dagster/dagster`

### Orchestration platform

- [Orchestration engine](https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster/_core): `python_modules/dagster/dagster/_core`
- [Scheduling and runs](https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster/_core): `python_modules/dagster/dagster/_core`
- [Kubernetes deployment](https://github.com/dagster-io/dagster/tree/master/helm/dagster/schema/schema/charts/dagster): `helm/dagster/schema/schema/charts/dagster`

### Control plane UI

- [Dagster web app](https://github.com/dagster-io/dagster/blob/master/js_modules/app-oss/src/App.tsx): `js_modules/app-oss/src/App.tsx`
- [UI application core](https://github.com/dagster-io/dagster/tree/master/js_modules/ui-core/src/app): `js_modules/ui-core/src/app`
- [Asset explorer](https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/asset-graph/AssetGraphExplorer.tsx): `js_modules/ui-core/src/asset-graph/AssetGraphExplorer.tsx`
- [Run and partition views](https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/partitions/usePartitionStepQuery.tsx): `js_modules/ui-core/src/partitions/usePartitionStepQuery.tsx`
- [Backfill details](https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/instance/backfill/useBackfillDetailsQuery.tsx): `js_modules/ui-core/src/instance/backfill/useBackfillDetailsQuery.tsx`
- [Launchpad sessions](https://github.com/dagster-io/dagster/blob/master/js_modules/ui-core/src/app/ExecutionSessionStorage.tsx): `js_modules/ui-core/src/app/ExecutionSessionStorage.tsx`

### Storage and integrations

- [Compute log storage](https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/cloud_storage_compute_log_manager.py): `python_modules/dagster/dagster/_core/storage/cloud_storage_compute_log_manager.py`
- [BigQuery pandas I/O](https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pandas/dagster_gcp_pandas/bigquery/bigquery_pandas_type_handler.py): `python_modules/libraries/dagster-gcp-pandas/dagster_gcp_pandas/bigquery/bigquery_pandas_type_handler.py`
- [BigQuery Spark I/O](https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pyspark/dagster_gcp_pyspark/bigquery/bigquery_pyspark_type_handler.py): `python_modules/libraries/dagster-gcp-pyspark/dagster_gcp_pyspark/bigquery/bigquery_pyspark_type_handler.py`

### Other components

- **Data developer**
- **Google BigQuery**
- **Browser state cache**

## Connections

- Data developer → Asset definitions: declares assets
- Asset definitions → Asset lineage graph: defines dependencies
- Asset definitions → Orchestration engine: submits definitions
- Orchestration engine → Scheduling and runs: plans and launches
- Scheduling and runs → Asset computation: executes
- Asset computation → BigQuery pandas I/O: uses I/O manager
- Asset computation → BigQuery Spark I/O: uses I/O manager
- BigQuery pandas I/O → Google BigQuery: reads and writes
- BigQuery Spark I/O → Google BigQuery: reads and writes
- Scheduling and runs → Compute log storage: captures logs
- Data developer → Dagster web app: opens control plane
- Dagster web app → UI application core: renders application
- UI application core → Asset explorer: shows assets
- UI application core → Run and partition views: queries partition runs
- UI application core → Backfill details: queries backfills
- UI application core → Launchpad sessions: persists launchpad state
- Launchpad sessions → Browser state cache: reads and writes
- Asset explorer → Asset lineage graph: visualizes lineage
- Kubernetes deployment → Orchestration engine: configures deployment