# tobikodata/sqlmesh architecture

> Architecture diagram of the GitHub repository tobikodata/sqlmesh, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tobikodata/sqlmesh
- Repository: https://github.com/tobikodata/sqlmesh

## Overview

##### Main Components Identification
The system consists of several key components that should be represented:

a) Core Components:
- Model Management System
- Virtual Data Environment Engine
- SQL Transformation Engine
- Testing Framework
- State Management System
- Scheduler System

b) Integration Layer:
- Multiple Database Adapters (Snowflake, BigQuery, DuckDB, etc.)
- CI/CD Integration (GitHub)
- Airflow Integration
- dbt Integration

c) User Interface Components:
- CLI Interface
- Web UI (React-based frontend)
- API Server (FastAPI backend)

**┌─ User Interface Layer ─────────────────────┐**
- Web UI (React)                          │
- CLI Interface                           │
- API Server (FastAPI)                    │
- ┬───────────────────────┘
- ↓
**┌─ Core Processing Layer ───────────────────┐**
- ┌─ Model Management ─┐ ┌─ Execution ────┐ │
- SQL Models      │ │ - Plan Builder  │ │
- Python Models   │ │ - Evaluator     │ │
- Seed Models     │ │ - Scheduler     │ │
- ┘ └────────────────┘ │
- ┌─ Virtual Environments ─────────────────┐ │
- Environment Management              │ │
- State Sync                         │ │
- Snapshot Management                │ │
- ┘ │
- ┬───────────────────────┘
- ↓
**┌─ Integration Layer ────────────────────────┐**
- ┌─ Database Adapters ┐ ┌─ External Tools ┐ │
- Snowflake       │ │ - GitHub CI/CD  │ │
- BigQuery        │ │ - Airflow       │ │
- DuckDB          │ │ - dbt           │ │
- ┘ └────────────────┘ │
- ┘

##### Additional Details

Important elements:
- Model lifecycle flow
- Virtual environment isolation boundaries
- Testing and validation workflows
- State synchronization paths
- SQL transformation pipeline

##### Key Relationships to Highlight

Interactions:
- Model definition to execution pipeline
- Virtual environment creation and management
- Integration points with external systems
- State synchronization between environments
- Testing and validation workflows

##### Notes for Implementation

- Clear separation between core and integration components
- Virtual environment isolation concept
- Demonstrate the SQL transformation pipeline
- Testing and validation workflows
- CI/CD integration points

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef ui fill:#3498db,stroke:#2980b9,color:white
    classDef core fill:#2ecc71,stroke:#27ae60,color:white
    classDef integration fill:#e67e22,stroke:#d35400,color:white
    classDef external fill:#95a5a6,stroke:#7f8c8d,color:white

    subgraph UI["User Interface Layer"]
        WebUI["Web UI (React)"]:::ui
        CLI["CLI Interface"]:::ui
        API["API Server (FastAPI)"]:::ui
    end

    subgraph Core["Core Processing Layer"]
        subgraph MM["Model Management"]
            SQLModels["SQL Models"]:::core
            PythonModels["Python Models"]:::core
            SeedModels["Seed Models"]:::core
        end

        subgraph EX["Execution"]
            PlanBuilder["Plan Builder"]:::core
            Evaluator["Evaluator"]:::core
            Scheduler["Scheduler"]:::core
        end

        subgraph VE["Virtual Environments"]
            EnvMgmt["Environment Management"]:::core
            StateSync["State Sync"]:::core
            SnapMgmt["Snapshot Management"]:::core
        end
    end

    subgraph INT["Integration Layer"]
        subgraph DB["Database Adapters"]
            Snowflake["Snowflake"]:::integration
            BigQuery["BigQuery"]:::integration
            DuckDB["DuckDB"]:::integration
        end

        subgraph EXT["External Tools"]
            GitHub["GitHub CI/CD"]:::integration
            Airflow["Airflow"]:::integration
            DBT["dbt"]:::integration
        end
    end

    %% Connections
    WebUI --> API
    CLI --> MM
    API --> MM
    
    MM --> EX
    EX --> VE
    VE --> DB
    VE --> EXT

    %% Click events
    click MM "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/core/model/"
    click VE "https://github.com/TobikoData/sqlmesh/blob/main/sqlmesh/core/environment.py"
    click EX "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/core/test/"
    click StateSync "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/core/state_sync/"
    click Scheduler "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/schedulers/"
    click DB "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/core/engine_adapter/"
    click GitHub "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/integrations/github/"
    click Airflow "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/schedulers/airflow/"
    click DBT "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/dbt/"
    click CLI "https://github.com/TobikoData/sqlmesh/tree/main/sqlmesh/cli/"
    click WebUI "https://github.com/TobikoData/sqlmesh/tree/main/web/client/"
    click API "https://github.com/TobikoData/sqlmesh/tree/main/web/server/"

    %% Legend
    subgraph Legend
        UI_L["User Interface"]:::ui
        Core_L["Core Component"]:::core
        Int_L["Integration"]:::integration
        Ext_L["External System"]:::external
    end
```
