How starlake-ai/starlake works
Project type and purpose
• This is a declarative, configuration-only ETL/ELT/Orchestration framework (“Starlake”).
• It provides a CLI core (Scala/Java), a Spark library, connector plugins (extractors, loaders, transformers), Python packages for orchestrators (Airflow, Dagster), plus template engines and DAG generators.
• Purpose: let analysts define extraction, loading, transformation, and orchestration pipelines via YAML without writing imperative code.
File structure
• Top-level artifacts:
• build.sbt / project/ → Scala build definitions (core engine)
• src/main/scala → core CLI, job definitions, connector factories (BigQuery, JDBC, Kafka, Snowflake, etc.)
• src/main/python/starlake-airflow, starlake-dagster, starlake-orchestration, starlake-snowflake → Python packages plug into orchestrators
• src/main/resources → DAG/templates, SQL templates, configuration defaults
• distrib & Dockerfile / scripts → distribution artifacts and Docker support
• Patterns:
• Plugin/Factory for connectors (extract/spi, sink, write-strategies, load-strategies)
• Template-driven DAG generation (Jinja)
• Separation between CLI, core engine, connector implementations, orchestrator adapters
README insights
• Core components: Extract, Load, Transform, Orchestrate
• YAML DSL drives pipeline definitions
• Supports multiple execution backends: Spark, CloudRun, Dataproc, Fargate, Shell
• Generates DAGs for Airflow, Dagster
Main components
a. CLI & Core Engine (Scala/Java)
b. Config Store (YAML files & templates)
c. Connector Layer:
• Extractors (JDBC, OpenAPI, GoogleSheets, file formats)
• Loaders (BigQuery, Snowflake, DuckDB, JDBC, Kafka, etc.)
• Transformers (SQL tasks, aggregation, merge strategies)
d. Template Engine (Jinja) & DAG Generator
e. Orchestration Adapters (Airflow, Dagster, native orchestrator via Python packages)
f. Execution Runners (Spark, Dataproc, Cloud Run, Fargate, Shell)
g. Target Systems (Data warehouses, file systems, message buses)
h. Distribution & Packaging (sbt, Python packaging, Docker)
Relationships & interactions
• User invokes CLI → reads YAML configs → Core Engine parses DSL → uses Connector Layer to:
1. Extract data from source via Extractor plugin → writes files to staging
2. Load staging files into target via Loader plugin (applying write-strategies)
3. Execute SQL transformation tasks via Transformer plugin
• Core Engine calls Template Engine → generates DAG definitions (Python DAG files)
• Orchestration Adapters (Python packages) register DAGs in orchestrators → schedule & run Execution Runners
• Execution Runners dispatch jobs to Spark, CloudRun, Dataproc, Fargate, or Shell
• Monitoring & lineage: Core Engine tracks dependencies and lineage in DAG templates
Architectural patterns & design principles
• Plugin/Factory: connector SPI for extract, load, transform
• Template Method: Jinja templates for DAG generation and SQL DDL/DDL mappings
• Adapter: Python packages adapt Core Engine to Airflow & Dagster
• Configuration-Driven: behaviors defined by YAML, minimal code changes
• Modular layering: CLI/API, core engine, connectors, orchestrator, execution
Essential elements
• Clear component names (e.g., “Extract SPI”, “Load SPI”, “Transform SPI”)
• Template resources and packaging (sbt, PyPI, Docker)
• Lineage feedback loop: after task execution, Core Engine updates metadata for downstream tasks