# starlake-ai/starlake architecture

> Architecture diagram of the GitHub repository starlake-ai/starlake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/starlake-ai/starlake
- Repository: https://github.com/starlake-ai/starlake

## Overview

##### Project type and purpose
- This is a declarative, configuration-only ETL/ELT/Orchestration framework (“Starlake”).
- It provides a CLI core (Scala/Java), a Spark library, connector plugins (extractors, loaders, transformers), Python packages for orchestrators (Airflow, Dagster), plus template engines and DAG generators.
- Purpose: let analysts define extraction, loading, transformation, and orchestration pipelines via YAML without writing imperative code.

##### File structure
- Top-level artifacts:
- build.sbt / project/ → Scala build definitions (core engine)
- src/main/scala → core CLI, job definitions, connector factories (BigQuery, JDBC, Kafka, Snowflake, etc.)
- src/main/python/starlake-airflow, starlake-dagster, starlake-orchestration, starlake-snowflake → Python packages plug into orchestrators
- src/main/resources → DAG/templates, SQL templates, configuration defaults
- distrib & Dockerfile / scripts → distribution artifacts and Docker support
- Patterns:
- Plugin/Factory for connectors (extract/spi, sink, write-strategies, load-strategies)
- Template-driven DAG generation (Jinja)
- Separation between CLI, core engine, connector implementations, orchestrator adapters

##### README insights
- Core components: Extract, Load, Transform, Orchestrate
- YAML DSL drives pipeline definitions
- Supports multiple execution backends: Spark, CloudRun, Dataproc, Fargate, Shell
- Generates DAGs for Airflow, Dagster

##### Main components
a. CLI & Core Engine (Scala/Java)
b. Config Store (YAML files & templates)
c. Connector Layer:
- Extractors (JDBC, OpenAPI, GoogleSheets, file formats)
- Loaders (BigQuery, Snowflake, DuckDB, JDBC, Kafka, etc.)
- Transformers (SQL tasks, aggregation, merge strategies)
d. Template Engine (Jinja) & DAG Generator
e. Orchestration Adapters (Airflow, Dagster, native orchestrator via Python packages)
f. Execution Runners (Spark, Dataproc, Cloud Run, Fargate, Shell)
g. Target Systems (Data warehouses, file systems, message buses)
h. Distribution & Packaging (sbt, Python packaging, Docker)

##### Relationships & interactions
- User invokes CLI → reads YAML configs → Core Engine parses DSL → uses Connector Layer to:
1. Extract data from source via Extractor plugin → writes files to staging
2. Load staging files into target via Loader plugin (applying write-strategies)
3. Execute SQL transformation tasks via Transformer plugin
- Core Engine calls Template Engine → generates DAG definitions (Python DAG files)
- Orchestration Adapters (Python packages) register DAGs in orchestrators → schedule & run Execution Runners
- Execution Runners dispatch jobs to Spark, CloudRun, Dataproc, Fargate, or Shell
- Monitoring & lineage: Core Engine tracks dependencies and lineage in DAG templates

##### Architectural patterns & design principles
- Plugin/Factory: connector SPI for extract, load, transform
- Template Method: Jinja templates for DAG generation and SQL DDL/DDL mappings
- Adapter: Python packages adapt Core Engine to Airflow & Dagster
- Configuration-Driven: behaviors defined by YAML, minimal code changes
- Modular layering: CLI/API, core engine, connectors, orchestrator, execution

##### Essential elements
- Clear component names (e.g., “Extract SPI”, “Load SPI”, “Transform SPI”)
- Template resources and packaging (sbt, PyPI, Docker)
- Lineage feedback loop: after task execution, Core Engine updates metadata for downstream tasks

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration & Templates
    subgraph "Configuration & Templates"
        ConfigStore["Config Store\n(YAML & Templates)"]:::config
    end

    %% Core Layer
    subgraph "Core (Scala/Java)"
        CLI["CLI & Core Engine"]:::core
        ExtractSPI["Extract SPI"]:::core
        ExtractImpl["Extractor Implementations"]:::core
        LoadSPI["Load SPI"]:::core
        LoadImpl["Loader Implementations"]:::core
        TransformSPI["Transform SPI"]:::core
    end

    %% Template Engine
    subgraph "Template Engine & DAG Generator"
        TemplateEngine["Template Engine\n& DAG Generator"]:::core
    end

    %% Orchestration Adapters
    subgraph "Python Orchestration Adapters"
        Airflow["Airflow Adapter"]:::python
        Dagster["Dagster Adapter"]:::python
        GenericOrch["Generic Orchestrator Adapter"]:::python
        SnowflakeOrch["Snowflake Orchestrator Adapter"]:::python
    end

    %% Execution Runners
    subgraph "Execution Runners"
        Spark["Spark"]:::external
        Dataproc["Dataproc"]:::external
        CloudRun["Cloud Run"]:::external
        Fargate["Fargate"]:::external
        Shell["Shell"]:::external
    end

    %% Target Systems
    subgraph "Target Systems"
        Targets["Data Warehouses,\nFile Systems,\nMessage Buses"]:::external
    end

    %% Distribution & Packaging
    subgraph "Distribution & Packaging"
        JVM["JVM Packaging & Docker"]:::pkg
        PyPkg["Python Packaging"]:::pkg
    end

    %% Data Flows
    ConfigStore -->|reads configs| CLI
    CLI -->|"parse DSL"| ExtractSPI
    ExtractSPI -->|"extract"| ExtractImpl
    CLI -->|"parse DSL"| LoadSPI
    LoadSPI -->|"load"| LoadImpl
    CLI -->|"parse DSL"| TransformSPI
    TransformSPI -->|"transform"| TransformSPI
    CLI -->|"parse DSL"| TemplateEngine
    TemplateEngine -->|"generate DAG"| Airflow
    TemplateEngine -->|"generate DAG"| Dagster
    TemplateEngine -->|"generate DAG"| GenericOrch
    TemplateEngine -->|"generate DAG"| SnowflakeOrch
    Airflow -->|"schedule"| Spark
    Dagster -->|"schedule"| Dataproc
    GenericOrch -->|"schedule"| CloudRun
    SnowflakeOrch -->|"schedule"| Fargate
    SnowflakeOrch -->|"schedule"| Shell
    Spark -->|"write data"| Targets
    Dataproc -->|"write data"| Targets
    CloudRun -->|"write data"| Targets
    Fargate -->|"write data"| Targets
    Shell -->|"write data"| Targets
    Spark -->|"update lineage"| CLI
    Dataproc -->|"update lineage"| CLI
    CloudRun -->|"update lineage"| CLI
    Fargate -->|"update lineage"| CLI
    Shell -->|"update lineage"| CLI

    %% Click Events
    click CLI "https://github.com/starlake-ai/starlake/tree/master/src/main/scala/ai/starlake"
    click ConfigStore "https://github.com/starlake-ai/starlake/tree/master/src/main/resources/templates"
    click ExtractSPI "https://github.com/starlake-ai/starlake/tree/master/src/main/scala/ai/starlake/extract/spi"
    click ExtractImpl "https://github.com/starlake-ai/starlake/tree/master/src/main/scala/ai/starlake/extract/impl"
    click LoadImpl "https://github.com/starlake-ai/starlake/tree/master/src/main/scala/ai/starlake/job/sink"
    click TransformSPI "https://github.com/starlake-ai/starlake/tree/master/src/main/scala/ai/starlake/job/transform"
    click TemplateEngine "https://github.com/starlake-ai/starlake/tree/master/src/main/scala/ai/starlake/schema/generator"
    click Airflow "https://github.com/starlake-ai/starlake/tree/master/src/main/python/starlake-airflow/ai/starlake/airflow"
    click Dagster "https://github.com/starlake-ai/starlake/tree/master/src/main/python/starlake-dagster/ai/starlake/dagster"
    click GenericOrch "https://github.com/starlake-ai/starlake/tree/master/src/main/python/starlake-orchestration/ai/starlake/orchestration"
    click SnowflakeOrch "https://github.com/starlake-ai/starlake/tree/master/src/main/python/starlake-snowflake/ai/starlake/snowflake"
    click JVM "https://github.com/starlake-ai/starlake/tree/master/distrib/"
    click PyPkg "https://github.com/starlake-ai/starlake/blob/master/src/main/python/starlake-airflow/setup.py"

    %% Styles
    classDef core fill:#D0E4F5,stroke:#0366D6,color:#0366D6;
    classDef python fill:#E6F5D0,stroke:#228B22,color:#228B22;
    classDef config fill:#F5E6D0,stroke:#D2691E,color:#D2691E;
    classDef external fill:#E0E0E0,stroke:#808080,color:#404040;
    classDef pkg fill:#F0D0F5,stroke:#800080,color:#800080;
```
