How maksimiliantsenkman/duckdb_orchestration works
Project type and purpose
• This is an open-source data pipeline orchestration tool written in Go.
• It processes templated SQL models with dependency resolution, executes them against DuckDB, writes Parquet outputs locally, and pushes them to Google Cloud Storage (GCS).
File structure
• Top-level Go source files and no subdirectories:
• main.go – entry point, parses flags, kicks off the pipeline
• graph.go – builds dependency graph from {{ ref(...) }} statements
• sql_parsing.go – handles template parsing and incremental logic
• refs.go – helper for resolving references in SQL models
• process_files.go – executes models in topological order, writes Parquet
• gcp_storage_ext.go – encapsulates GCS upload/download extension
• sql_parsing.go & process_files.go – implement the core pipeline loop
• go.mod/go.sum – external dependencies (DuckDB Go driver, GCS SDK)
Key insights from README
• Dependency graph + topological sort pattern
• Incremental vs full-refresh templating
• DuckDB as embedded SQL engine
• GCS integration for output storage
• Concurrent processing with configurable threads
Main components
a. Pipeline Orchestrator (main.go)
• Flag parsing (threads, full-refresh)
• Orchestration controller
b. Dependency Resolver (graph.go, refs.go)
• Scans MODELS_FOLDER for SQL files
• Builds directed acyclic graph (DAG) based on {{ ref("...") }}
c. SQL Template Engine (sql_parsing.go)
• Renders templates, applies is_incremental() logic
d. Execution Engine (process_files.go + DuckDB)
• Executes rendered SQL in topological order
• Writes query results to Parquet
e. Storage Layer
• Local disk storage (OUTPUT_FOLDER)
• GCS Integration (gcp_storage_ext.go)
f. Configuration & Metadata
• .env loader: DUCK_DB_FILE, MAPPINGS_PATH, etc.
• Mappings JSON parsed at startup
Relationships and data flow
• User invokes CLI → Orchestrator
• Orchestrator reads config → loads mappings, env vars
• Dependency Resolver scans models, emits DAG → ordered list
• SQL Template Engine renders each model → SQL statements
• Execution Engine runs SQL on DuckDB → Parquet files
• Storage Layer uploads Parquet → GCS bucket per model
Architectural patterns & principles
• Modular layering: separation of concerns (parsing, graphing, execution, storage)
• DAG-based execution for dependency management
• Concurrency via worker threads controlled by orchestrator
• Template-driven configurability (Jinja-style ref & is_incremental)
Relevant technologies & libraries
• Go 1.20+
• DuckDB embedded SQL engine (Go driver)
• Google Cloud Storage SDK for Go
• Parquet file writer (compression, partitioning)