# dagster-io/dagster-open-platform architecture

> Architecture diagram of the GitHub repository dagster-io/dagster-open-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dagster-io/dagster-open-platform
- Repository: https://github.com/dagster-io/dagster-open-platform

## Overview

##### Main Components

a. Data Sources Layer:
- External Services Integration
- GitHub (dlt/sources/github)
- Buildkite (dlt/sources/buildkite)
- HubSpot (dlt/sources/hubspot)
- Thinkific (dlt/sources/thinkific)
- Stripe (stripe/)
- PyPI (pypi/)
- AWS Services (aws/)
- Segment (segment/)
- Scout (scout/)
- Sling (sling/)

b. Data Integration Layer:
- ETL/ELT Tools
- Fivetran (fivetran/)
- DLT (Data Load Tool) (dlt/)
- Hightouch (hightouch/)

c. Data Warehouse Layer:
- Storage Systems
- Snowflake (snowflake/)

d. Transformation Layer:
- DBT Integration (dbt/)
- Models
- Snapshots
- Macros
- Tests

**[External Data Sources]**
- ↓
**[Data Integration Layer]**
- ↓
**[Data Warehouse]**
- ↓
**[Transformation Layer]**
- ↓
**[Analytics & Operations]**

##### Key Relationships to Highlight

- Orchestration role of Dagster across all components
- Data flow patterns between layers
- Scheduled jobs and sensors with distinct symbols

##### Architectural Patterns to Emphasize

- Modular design (separate directories for each integration)
- Resource abstraction (resources.py files)
- Schedule management (schedules.py files)
- Asset organization (assets.py files)

##### Additional Elements

- Annotations for key configuration points
- Critical paths and dependencies
- Error handling and monitoring points

##### Technical Details

- Asset definitions and dependencies
- Schedule and partition configurations
- Resource requirements and configurations
- Integration points with external systems

Remember to include:
- Version control integration points
- Testing frameworks
- Monitoring and observability components
- Error handling and recovery paths
- Data quality check points

This system design should emphasize Dagster's role as the central orchestrator while clearly showing how it integrates with various data sources, transformation tools, and storage systems in a production environment.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef source fill:#a8d1f0,stroke:#333
    classDef integration fill:#90EE90,stroke:#333
    classDef warehouse fill:#FFA500,stroke:#333
    classDef transform fill:#DDA0DD,stroke:#333
    classDef analytics fill:#FFB6C1,stroke:#333

    %% Data Sources Layer
    subgraph Sources["Data Sources Layer"]
        GitHub["GitHub"]:::source
        Buildkite["Buildkite"]:::source
        HubSpot["HubSpot"]:::source
        Thinkific["Thinkific"]:::source
        Stripe["Stripe"]:::source
        PyPI["PyPI"]:::source
        AWS["AWS Services"]:::source
        Segment["Segment"]:::source
        Scout["Scout"]:::source
        Sling["Sling"]:::source
    end

    %% Data Integration Layer
    subgraph Integration["Data Integration Layer"]
        Fivetran["Fivetran"]:::integration
        DLT["DLT"]:::integration
        Hightouch["Hightouch"]:::integration
    end

    %% Data Warehouse Layer
    subgraph Warehouse["Data Warehouse Layer"]
        Snowflake["Snowflake"]:::warehouse
    end

    %% Transformation Layer
    subgraph Transform["Transformation Layer"]
        DBT["DBT Core"]:::transform
        DBTInt["DBT Integration"]:::transform
        Snapshots["DBT Snapshots"]:::transform
        Macros["DBT Macros"]:::transform
        Tests["DBT Tests"]:::transform
    end

    %% Analytics Layer
    subgraph Analytics["Analytics & Operations"]
        Reports["Reporting"]:::analytics
        Monitoring["Monitoring"]:::analytics
    end

    %% Relationships
    Sources --> Integration
    Integration --> Warehouse
    Warehouse --> Transform
    Transform --> Analytics

    %% Click events for component mapping
    click GitHub "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/dlt/sources/github/"
    click Buildkite "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/dlt/sources/buildkite/"
    click HubSpot "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/dlt/sources/hubspot/"
    click Thinkific "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/dlt/sources/thinkific/"
    click Stripe "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/stripe/"
    click PyPI "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/pypi/"
    click AWS "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/aws/"
    click Segment "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/segment/"
    click Scout "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/scout/"
    click Sling "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/sling/"
    click Fivetran "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/fivetran/"
    click DLT "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/dlt/"
    click Hightouch "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/hightouch/"
    click Snowflake "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/snowflake/"
    click DBT "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform_dbt/"
    click DBTInt "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/dbt/"
    click Snapshots "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform_dbt/snapshots/"
    click Macros "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform_dbt/macros/"
    click Tests "https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform_dbt/tests/"

    %% Legend
    subgraph Legend
        SourceLegend["Data Sources"]:::source
        IntegrationLegend["Integration Tools"]:::integration
        WarehouseLegend["Data Warehouse"]:::warehouse
        TransformLegend["Transformation"]:::transform
        AnalyticsLegend["Analytics"]:::analytics
    end
```
