# orchestra-hq/orchestra-blueprints architecture

> Architecture diagram of the GitHub repository orchestra-hq/orchestra-blueprints, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/orchestra-hq/orchestra-blueprints
- Repository: https://github.com/orchestra-hq/orchestra-blueprints

## Overview

orchestra-blueprints is a collection of reference implementations for Orchestra-driven data platforms, not one integrated application. Its principal workflow is to define or trigger data pipelines, extract from external systems, load warehouse data, and optionally publish lineage or analyze platform usage. The README also documents warehouse-specific dbt projects, Azure ML assets, metadata ingestion, and reusable orchestration patterns. Some documented areas are not represented in the supplied file tree or sampled source, so their internal wiring is not inferred.

Main entry points: [`python/dlt/hubspot_pipeline.py`](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/hubspot_pipeline.py) (HubSpot pipeline), [`python/dlt/linear_pipeline.py`](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/linear_pipeline.py) (Linear pipelines), [`python/dlt/google_sheets_pipeline.py`](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/google_sheets_pipeline.py) (Google Sheets pipeline), [`python/google_sheets_native/google_sheet_ingest.py`](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/google_sheets_native/google_sheet_ingest.py) (Native Sheets loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration patterns"]
  node_multi_runner["Multi-pipeline runner"]
  node_runner_logic["Runner API client"]
  node_savings["Warehouse savings analysis<br/>[graph.py]"]
end

subgraph group_ingestion["Data ingestion"]
  node_dlt_examples["dlt ingestion examples"]
  node_hubspot["HubSpot pipeline"]
  node_linear["Linear pipelines<br/>[linear_pipeline.py]"]
  node_sheets["Google Sheets pipeline"]
  node_sheets_native["Native Sheets loader"]
end

subgraph group_warehouse["Warehouse projects"]
  node_dbt_projects["Warehouse dbt projects"]
end

subgraph group_lineage["Platform lineage"]
  node_lineage_extractors["Lineage extractors<br/>[bigquery.py]"]
  node_lineage_publish["Lineage publisher<br/>[publish_lineage.py]"]
  node_metadata_store[("Landed metadata")]
end

subgraph group_workers["ML and workers"]
  node_azure_ml["Azure ML pipeline<br/>[pipeline_script.py]"]
  node_prep["Data preparation<br/>[prep_data.py]"]
  node_train["Model training<br/>[train.py]"]
  node_metadata_api["Metadata API example<br/>[run.py]"]
end

node_operator(("Platform operator"))
node_orchestra{{"Orchestra platform"}}
node_source_services(("SaaS and Sheets"))
node_dlt_runtime{{"dlt runtime"}}
node_bigquery[("BigQuery destination")]
node_motherduck[("MotherDuck destination")]

node_operator -->|"runs"| node_multi_runner
node_multi_runner -->|"delegates"| node_runner_logic
node_runner_logic -->|"starts runs"| node_orchestra
node_operator -->|"runs"| node_hubspot
node_hubspot -->|"extracts"| node_source_services
node_hubspot -->|"loads via"| node_dlt_runtime
node_dlt_runtime -->|"writes data"| node_bigquery
node_operator -->|"runs"| node_linear
node_linear -->|"extracts"| node_source_services
node_linear -->|"loads via"| node_dlt_runtime
node_dlt_runtime -->|"writes data"| node_motherduck
node_operator -->|"runs"| node_sheets
node_sheets -->|"extracts"| node_source_services
node_sheets -->|"loads via"| node_dlt_runtime
node_dlt_runtime -->|"writes data"| node_bigquery
node_operator -->|"runs"| node_sheets_native
node_sheets_native -->|"reads sheet"| node_source_services
node_sheets_native -->|"loads rows"| node_bigquery
node_operator -.->|"uses"| node_dbt_projects
node_operator -->|"runs"| node_lineage_extractors
node_lineage_extractors -->|"queries metadata"| node_bigquery
node_lineage_extractors -->|"loads extracts"| node_dlt_runtime
node_dlt_runtime -->|"lands metadata"| node_metadata_store
node_lineage_publish -->|"queries"| node_metadata_store
node_lineage_publish -->|"publishes assets"| node_orchestra
node_operator -->|"submits"| node_azure_ml
node_azure_ml -->|"dispatches"| node_prep
node_azure_ml -->|"dispatches"| node_train
node_prep -->|"passes prepared data"| node_train
node_operator -->|"runs"| node_metadata_api
node_dlt_examples -.->|"uses"| node_dlt_runtime
node_operator -.->|"uses"| node_savings
node_savings -.->|"analyzes API"| node_orchestra

click node_multi_runner "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/patterns/run_multiple_pipelines/run_multiple_pipelines.py"
click node_runner_logic "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/patterns/run_multiple_pipelines/_pipeline_runner.py"
click node_savings "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/patterns/warehouse_savings/graph.py"
click node_dlt_examples "https://github.com/orchestra-hq/orchestra-blueprints/tree/main/python/dlt"
click node_hubspot "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/hubspot_pipeline.py"
click node_linear "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/linear_pipeline.py"
click node_sheets "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/google_sheets_pipeline.py"
click node_sheets_native "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/google_sheets_native/google_sheet_ingest.py"
click node_dbt_projects "https://github.com/orchestra-hq/orchestra-blueprints/tree/main/dbt_projects"
click node_lineage_extractors "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/lineage/sources/bigquery.py"
click node_lineage_publish "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/lineage/publish_lineage.py"
click node_metadata_store "https://github.com/orchestra-hq/orchestra-blueprints/tree/main/python/lineage"
click node_azure_ml "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/azure/ml/pipeline_script.py"
click node_prep "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/azure/ml/scripts/prep_data.py"
click node_train "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/azure/ml/scripts/train.py"
click node_metadata_api "https://github.com/orchestra-hq/orchestra-blueprints/blob/main/metadata_api/run.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_multi_runner,node_runner_logic,node_savings toneBlue
class node_dlt_examples,node_hubspot,node_linear,node_sheets,node_sheets_native,node_bigquery,node_motherduck toneAmber
class node_dbt_projects toneMint
class node_lineage_extractors,node_lineage_publish,node_metadata_store toneRose
class node_azure_ml,node_prep,node_train,node_metadata_api,node_operator,node_orchestra,node_source_services,node_dlt_runtime toneIndigo
```

## Components

### Orchestration patterns

- [Multi-pipeline runner](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/patterns/run_multiple_pipelines/run_multiple_pipelines.py): `patterns/run_multiple_pipelines/run_multiple_pipelines.py`
- [Runner API client](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/patterns/run_multiple_pipelines/_pipeline_runner.py): `patterns/run_multiple_pipelines/_pipeline_runner.py`
- [Warehouse savings analysis](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/patterns/warehouse_savings/graph.py): `patterns/warehouse_savings/graph.py`

### Data ingestion

- [dlt ingestion examples](https://github.com/orchestra-hq/orchestra-blueprints/tree/main/python/dlt): `python/dlt`
- [HubSpot pipeline](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/hubspot_pipeline.py): `python/dlt/hubspot_pipeline.py`
- [Linear pipelines](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/linear_pipeline.py): `python/dlt/linear_pipeline.py`
- [Google Sheets pipeline](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/dlt/google_sheets_pipeline.py): `python/dlt/google_sheets_pipeline.py`
- [Native Sheets loader](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/google_sheets_native/google_sheet_ingest.py): `python/google_sheets_native/google_sheet_ingest.py`

### Warehouse projects

- [Warehouse dbt projects](https://github.com/orchestra-hq/orchestra-blueprints/tree/main/dbt_projects): `dbt_projects`

### Platform lineage

- [Lineage extractors](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/lineage/sources/bigquery.py): `python/lineage/sources/bigquery.py`
- [Lineage publisher](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/lineage/publish_lineage.py): `python/lineage/publish_lineage.py`
- [Landed metadata](https://github.com/orchestra-hq/orchestra-blueprints/tree/main/python/lineage): `python/lineage`

### ML and workers

- [Azure ML pipeline](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/azure/ml/pipeline_script.py): `python/azure/ml/pipeline_script.py`
- [Data preparation](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/azure/ml/scripts/prep_data.py): `python/azure/ml/scripts/prep_data.py`
- [Model training](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/python/azure/ml/scripts/train.py): `python/azure/ml/scripts/train.py`
- [Metadata API example](https://github.com/orchestra-hq/orchestra-blueprints/blob/main/metadata_api/run.py): `metadata_api/run.py`

### Other components

- **Platform operator**
- **Orchestra platform**
- **SaaS and Sheets**
- **dlt runtime**
- **BigQuery destination**
- **MotherDuck destination**

## Connections

- Platform operator → Multi-pipeline runner: runs
- Multi-pipeline runner → Runner API client: delegates
- Runner API client → Orchestra platform: starts runs
- Platform operator → HubSpot pipeline: runs
- HubSpot pipeline → SaaS and Sheets: extracts
- HubSpot pipeline → dlt runtime: loads via
- dlt runtime → BigQuery destination: writes data
- Platform operator → Linear pipelines: runs
- Linear pipelines → SaaS and Sheets: extracts
- Linear pipelines → dlt runtime: loads via
- dlt runtime → MotherDuck destination: writes data
- Platform operator → Google Sheets pipeline: runs
- Google Sheets pipeline → SaaS and Sheets: extracts
- Google Sheets pipeline → dlt runtime: loads via
- dlt runtime → BigQuery destination: writes data
- Platform operator → Native Sheets loader: runs
- Native Sheets loader → SaaS and Sheets: reads sheet
- Native Sheets loader → BigQuery destination: loads rows
- Platform operator → Warehouse dbt projects: uses
- Platform operator → Lineage extractors: runs
- Lineage extractors → BigQuery destination: queries metadata
- Lineage extractors → dlt runtime: loads extracts
- dlt runtime → Landed metadata: lands metadata
- Lineage publisher → Landed metadata: queries
- Lineage publisher → Orchestra platform: publishes assets
- Platform operator → Azure ML pipeline: submits
- Azure ML pipeline → Data preparation: dispatches
- Azure ML pipeline → Model training: dispatches
- Data preparation → Model training: passes prepared data
- Platform operator → Metadata API example: runs
- dlt ingestion examples → dlt runtime: uses
- Platform operator → Warehouse savings analysis: uses
- Warehouse savings analysis → Orchestra platform: analyzes API