How dagster-io/dagster works
Dagster is a cloud-native data asset and pipeline orchestrator. The principal workflow is to declare assets, have Dagster schedule and execute their computations, persist results, and inspect lineage and run health through its control-plane UI. The tree also exposes deployment configuration, BigQuery integrations, and UI support for runs, partitions, backfills, and cached queries. The supplied source excerpts are selective, so links to unsampled runtime subsystems are included only where the README documents the workflow; detailed internal wiring is not inferred.
Main entry points: app UI application core · dagster Asset definitions · _core Scheduling and runs · App.tsx Dagster web app
Components 18
Definitions and execution
- Asset definitions
python_modules/dagster/dagster - Asset lineage graph
js_modules/ui-core/src/asset-graph - Asset computation
python_modules/dagster/dagster
Orchestration platform
- Orchestration engine
python_modules/dagster/dagster/_core - Scheduling and runs
python_modules/dagster/dagster/_core - Kubernetes deployment
helm/dagster/schema/schema/charts/dagster
Control plane UI
- Dagster web app
js_modules/app-oss/src/App.tsx - UI application core
js_modules/ui-core/src/app - Asset explorer
js_modules/ui-core/src/asset-graph/AssetGraphExplorer.tsx - Run and partition views
js_modules/ui-core/src/partitions/usePartitionStepQuery.tsx - Backfill details
js_modules/ui-core/src/instance/backfill/useBackfillDetailsQuery.tsx - Launchpad sessions
js_modules/ui-core/src/app/ExecutionSessionStorage.tsx
Storage and integrations
- Compute log storage
python_modules/dagster/dagster/_core/storage/cloud_storage_compute_log_manager.py - BigQuery pandas I/O
python_modules/libraries/dagster-gcp-pandas/dagster_gcp_pandas/bigquery/bigquery_pandas_type_handler.py - BigQuery Spark I/O
python_modules/libraries/dagster-gcp-pyspark/dagster_gcp_pyspark/bigquery/bigquery_pyspark_type_handler.py
Other components
- Data developer
- Google BigQuery
- Browser state cache
Connections 19
- Data developer → Asset definitions: declares assets
- Asset definitions → Asset lineage graph: defines dependencies
- Asset definitions → Orchestration engine: submits definitions
- Orchestration engine → Scheduling and runs: plans and launches
- Scheduling and runs → Asset computation: executes
- Asset computation → BigQuery pandas I/O: uses I/O manager
- Asset computation → BigQuery Spark I/O: uses I/O manager
- BigQuery pandas I/O → Google BigQuery: reads and writes
- BigQuery Spark I/O → Google BigQuery: reads and writes
- Scheduling and runs → Compute log storage: captures logs
- Data developer → Dagster web app: opens control plane
- Dagster web app → UI application core: renders application
- UI application core → Asset explorer: shows assets
- UI application core → Run and partition views: queries partition runs
- UI application core → Backfill details: queries backfills
- UI application core → Launchpad sessions: persists launchpad state
- Launchpad sessions → Browser state cache: reads and writes
- Asset explorer → Asset lineage graph: visualizes lineage
- Kubernetes deployment → Orchestration engine: configures deployment