# snowflake-labs/schemachange architecture

> Architecture diagram of the GitHub repository snowflake-labs/schemachange, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/snowflake-labs/schemachange
- Repository: https://github.com/snowflake-labs/schemachange

## Overview

##### Project Type and Purpose
- This is a Python-based CLI tool (“schemachange”) for managing Snowflake database schema changes (a database migration utility).
- It follows an Imperative DCM (Database Change Management) approach, inspired by Flyway, and uses file-based SQL change scripts (versioned, repeatable, always).
- Key use cases: scanning a scripts folder tree, rendering and deploying scripts via Jinja templating, recording applied changes in a Snowflake change history table.

##### File Structure Analysis
- Top-level:
- schemachange/          → main package with modules:
– cli.py               → entrypoint, command parsing (deploy, render).
– config/              → YAML & CLI args parsing, config merging, secrets filtering.
– session/             → Snowflake connection abstractions (SnowflakeSession, Script wrapper, Credential).
– JinjaTemplateProcessor.py & JinjaEnvVar.py → template engine integration.
– deploy.py            → orchestrates the deploy command.
– redact_config_secrets.py → filters secrets in logs.
- demo/                  → sample change-script projects (basics, citibike, Jinja demos).
- tests/                 → pytest suite covering config, session, template processing, CLI misc, main.
- Dockerfile, workflows  → CI/CD, packaging, publishing.
- Config files: pyproject.toml, setup.cfg, requirements.txt, .pre-commit, GitHub Actions YAMLs.

##### README Insights
- Explains architecture components: change scripts folder, Jinja support with modules folder, connections.toml for Snowflake credentials.
- Describes two main commands: deploy (scans root-folder, processes versioned/repeatable/always scripts, updates metadata table) and render (renders a single script).
- Shows DevOps flow: packaging as pip or Docker image, integrating with CI/CD.

a. Main Components
- CLI Entrypoint (schemachange/cli.py)
- Configuration Loader (YAML + CLI args + environment variables)
- Template Processor (Jinja engine + custom filters/env_var)
- ChangeScript Scanner & Parser (file system walker reading .sql/.sql.jinja scripts)
- Deployment Orchestrator (deploy.py): orders scripts (versioned → repeatable → always)
- SnowflakeSession Adapter: wraps Snowflake Python Connector
- ChangeHistory Table (Snowflake): METADATA.SCHEMACHANGE.CHANGE_HISTORY
- External Dependencies: Snowflake Python Connector, Jinja2, PyYAML, Pandas
- Demo Projects (on-disk scripts) and Modules Folder (optional Jinja macros)

b. Relationships & Data Flows
1. User invokes CLI (deploy or render) → cli.py parses commands.
2. Config Loader merges CLI args, schemachange-config.yml, connections.toml, env vars.
3. For deploy:
– File Scanner reads scripts under root-folder → yields ordered list.
– Template Processor applies Jinja templating (uses modules-folder for macros).
– Deployment Orchestrator invokes SnowflakeSession for each script:
· Connect to Snowflake → execute “USE” statements → run SQL.
· After execution, reset session context.
· Write metadata to ChangeHistory table.
4. For render:
– File Scanner locates specific script → Template Processor renders to console.
5. SnowflakeSession uses Snowflake Connector to reach Snowflake cloud service.

c. Architectural Patterns & Principles
- Layered architecture: CLI → Config → Processing → Persistence (Snowflake)
- Dependency Injection: passing connection, config, and template engine into orchestrator
- Single Responsibility: each module (config parser, template processor, session adapter) has a narrow focus
- Plugin-style extensibility: modules-folder for user-defined Jinja macros
- Idempotent, ordered migration pattern (versioned/repeatable/always)

d. Technologies & Frameworks
- Python 3.x
- Jinja2 for templating
- PyYAML for config
- Snowflake Python Connector (snowflake-connector-python)
- pytest for testing
- Docker for containerized runs
- GitHub Actions & tox for CI

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Subgraph: schemachange Package
    subgraph "schemachange Package"
        direction TB
        CLI["CLI Entrypoint<br/>(cli.py)"]:::internal
        ParseCLI["CLI Args Parser<br/>(parse_cli_args.py)"]:::internal
        MergedConfig["Config Merger<br/>(get_merged_config.py)"]:::internal
        BaseConfig["BaseConfig"]:::internal
        DeployConfig["DeployConfig"]:::internal
        RenderConfig["RenderConfig"]:::internal
        ChangeHistoryConfig["ChangeHistoryTable Config"]:::internal
        Utils["Config Utilities"]:::internal
        SecretsRedactor["Secrets Redactor<br/>(redact_config_secrets.py)"]:::internal
        CoreEngine["Jinja Template Processor<br/>(JinjaTemplateProcessor.py)"]:::internal
        EnvVar["Env Var Filter<br/>(JinjaEnvVar.py)"]:::internal
        ScriptScanner["Script Scanner<br/>(session/Script.py)"]:::internal
        SnowSession["SnowflakeSession Adapter<br/>(SnowflakeSession.py)"]:::internal
        Credential["Credentials Model<br/>(Credential.py)"]:::internal
        Orchestrator["Deployment Orchestrator<br/>(deploy.py)"]:::internal
    end

    %% Subgraph: Demo Scripts
    subgraph "Demo Change-Scripts Folder" 
        direction TB
        DemoBasics["basics_demo"]:::external
        DemoCitibike["citibike_demo"]:::external
        DemoJinja["citibike_demo_jinja<br/>(modules folder)"]:::external
    end

    %% Subgraph: CI/CD & Packaging
    subgraph "CI/CD & Packaging"
        direction TB
        Dockerfile["Dockerfile"]:::thirdparty
        Workflow["GitHub Actions Workflow"]:::thirdparty
        Pyproject["pyproject.toml<br/>setup.cfg<br/>requirements.txt"]:::thirdparty
    end

    %% External Systems
    FS["File System<br/>(.sql/.jinja files)"]:::external
    SnowflakeDB["Snowflake Service"]:::external
    ChangeHistoryTable["METADATA.SCHEMACHANGE.CHANGE_HISTORY"]:::external

    %% Connections within schemachange Package
    CLI --> ParseCLI
    CLI -->|"merge config"| MergedConfig
    ParseCLI --> MergedConfig
    MergedConfig --> BaseConfig
    MergedConfig --> DeployConfig
    MergedConfig --> RenderConfig
    MergedConfig --> ChangeHistoryConfig
    MergedConfig --> Utils
    MergedConfig --> SecretsRedactor

    CLI -->|"deploy"| Orchestrator
    CLI -->|"render"| ScriptScanner

    %% Deploy flow
    Orchestrator --> ScriptScanner
    Orchestrator --> CoreEngine
    Orchestrator --> SnowSession

    %% Render flow
    ScriptScanner --> CoreEngine

    %% Template dependencies
    CoreEngine --> EnvVar
    DemoJinja --> CoreEngine

    %% File system scan
    FS --> ScriptScanner
    DemoBasics --> FS
    DemoCitibike --> FS
    DemoJinja --> FS

    %% Snowflake interaction
    SnowSession --> Credential
    SnowSession --> SnowflakeDB
    SnowSession --> ChangeHistoryTable

    %% CI/CD pipeline connections
    Pyproject --> Dockerfile
    Pyproject --> Workflow

    %% Click events
    click CLI "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/cli.py"
    click ParseCLI "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/config/parse_cli_args.py"
    click MergedConfig "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/config/get_merged_config.py"
    click BaseConfig "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/config/BaseConfig.py"
    click DeployConfig "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/config/DeployConfig.py"
    click RenderConfig "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/config/RenderConfig.py"
    click ChangeHistoryConfig "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/config/ChangeHistoryTable.py"
    click Utils "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/config/utils.py"
    click SecretsRedactor "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/redact_config_secrets.py"
    click CoreEngine "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/JinjaTemplateProcessor.py"
    click EnvVar "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/JinjaEnvVar.py"
    click ScriptScanner "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/session/Script.py"
    click SnowSession "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/session/SnowflakeSession.py"
    click Credential "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/session/Credential.py"
    click Orchestrator "https://github.com/snowflake-labs/schemachange/blob/master/schemachange/deploy.py"
    click DemoBasics "https://github.com/snowflake-labs/schemachange/tree/master/demo/basics_demo"
    click DemoCitibike "https://github.com/snowflake-labs/schemachange/tree/master/demo/citibike_demo"
    click DemoJinja "https://github.com/snowflake-labs/schemachange/tree/master/demo/citibike_demo_jinja"
    click Dockerfile "https://github.com/snowflake-labs/schemachange/tree/master/Dockerfile"
    click Workflow "https://github.com/snowflake-labs/schemachange/blob/master/.github/workflows/python-publish.yml"
    click Pyproject "https://github.com/snowflake-labs/schemachange/blob/master/pyproject.toml"

    %% Styles
    classDef internal fill:#cce5ff,stroke:#004085,color:#004085
    classDef external fill:#d4edda,stroke:#155724,color:#155724
    classDef thirdparty fill:#fff3cd,stroke:#856404,color:#856404
```
