# maksimiliantsenkman/duckdb_orchestration architecture

> Architecture diagram of the GitHub repository maksimiliantsenkman/duckdb_orchestration, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maksimiliantsenkman/duckdb_orchestration
- Repository: https://github.com/maksimiliantsenkman/duckdb_orchestration

## Overview

##### Project type and purpose
- This is an open-source data pipeline orchestration tool written in Go.
- It processes templated SQL models with dependency resolution, executes them against DuckDB, writes Parquet outputs locally, and pushes them to Google Cloud Storage (GCS).

##### File structure
- Top-level Go source files and no subdirectories:
- main.go – entry point, parses flags, kicks off the pipeline
- graph.go – builds dependency graph from `{{ ref(...) }}` statements
- sql_parsing.go – handles template parsing and incremental logic
- refs.go – helper for resolving references in SQL models
- process_files.go – executes models in topological order, writes Parquet
- gcp_storage_ext.go – encapsulates GCS upload/download extension
- sql_parsing.go & process_files.go – implement the core pipeline loop
- go.mod/go.sum – external dependencies (DuckDB Go driver, GCS SDK)

##### Key insights from README
- Dependency graph + topological sort pattern
- Incremental vs full-refresh templating
- DuckDB as embedded SQL engine
- GCS integration for output storage
- Concurrent processing with configurable threads

##### Main components
a. Pipeline Orchestrator (main.go)
- Flag parsing (threads, full-refresh)
- Orchestration controller
b. Dependency Resolver (graph.go, refs.go)
- Scans `MODELS_FOLDER` for SQL files
- Builds directed acyclic graph (DAG) based on `{{ ref("...") }}`
c. SQL Template Engine (sql_parsing.go)
- Renders templates, applies `is_incremental()` logic
d. Execution Engine (process_files.go + DuckDB)
- Executes rendered SQL in topological order
- Writes query results to Parquet
e. Storage Layer
- Local disk storage (OUTPUT_FOLDER)
- GCS Integration (gcp_storage_ext.go)
f. Configuration & Metadata
- `.env` loader: DUCK_DB_FILE, MAPPINGS_PATH, etc.
- Mappings JSON parsed at startup

##### Relationships and data flow
- User invokes CLI → Orchestrator
- Orchestrator reads config → loads mappings, env vars
- Dependency Resolver scans models, emits DAG → ordered list
- SQL Template Engine renders each model → SQL statements
- Execution Engine runs SQL on DuckDB → Parquet files
- Storage Layer uploads Parquet → GCS bucket per model

##### Architectural patterns & principles
- Modular layering: separation of concerns (parsing, graphing, execution, storage)
- DAG-based execution for dependency management
- Concurrency via worker threads controlled by orchestrator
- Template-driven configurability (Jinja-style `ref` & `is_incremental`)

##### Relevant technologies & libraries
- Go 1.20+
- DuckDB embedded SQL engine (Go driver)
- Google Cloud Storage SDK for Go
- Parquet file writer (compression, partitioning)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration & Metadata"
        ConfigLoader["Config & Mappings Loader"]:::config
    end

    subgraph "Pipeline Orchestrator"
        Orchestrator["CLI Orchestrator<br/>(threads, full-refresh)"]:::cli
    end

    subgraph "Dependency Resolver"
        DAG["DAG Builder"]:::processing
        Refs["Ref Resolver"]:::processing
    end

    subgraph "SQL Template Engine"
        Template["Template Renderer<br/>(is_incremental)"]:::processing
    end

    subgraph "Execution Engine"
        Executor["Executor<br/>(DuckDB & Parquet)"]:::processing
        DuckDB["DuckDB Engine"]:::external
    end

    subgraph "Storage Layer"
        Local["Local Disk<br/>(Parquet)"]:::storage
        GCS["GCS Bucket"]:::external
    end

    ConfigLoader -->|"load .env & mappings.json"| Orchestrator
    Orchestrator -->|"scan models"| DAG
    Orchestrator -->|"resolve refs"| Refs
    DAG -->|"DAG definition"| Template
    Refs -->|"ref resolution"| Template
    Template -->|"rendered SQL"| Executor
    Executor -->|"exec SQL"| DuckDB
    Executor -->|"query results"| Local
    Local -->|"upload Parquet"| GCS

    click ConfigLoader "https://github.com/maksimiliantsenkman/duckdb_orchestration/blob/main/go.mod"
    click Orchestrator "https://github.com/maksimiliantsenkman/duckdb_orchestration/blob/main/main.go"
    click DAG "https://github.com/maksimiliantsenkman/duckdb_orchestration/blob/main/graph.go"
    click Refs "https://github.com/maksimiliantsenkman/duckdb_orchestration/blob/main/refs.go"
    click Template "https://github.com/maksimiliantsenkman/duckdb_orchestration/blob/main/sql_parsing.go"
    click Executor "https://github.com/maksimiliantsenkman/duckdb_orchestration/blob/main/process_files.go"
    click GCS "https://github.com/maksimiliantsenkman/duckdb_orchestration/blob/main/gcp_storage_ext.go"

    classDef cli fill:#bfdbfe,stroke:#1e3a8a,stroke-width:2px
    classDef processing fill:#d1fae5,stroke:#065f46,stroke-width:2px
    classDef external fill:#fed7aa,stroke:#b45309,stroke-width:2px
    classDef storage fill:#fef3c7,stroke:#a16207,stroke-width:2px
    classDef config fill:#fef3c7,stroke:#92400e,stroke-width:2px
```
