How snowflake-labs/schemachange works
Project Type and Purpose
• This is a Python-based CLI tool (“schemachange”) for managing Snowflake database schema changes (a database migration utility).
• It follows an Imperative DCM (Database Change Management) approach, inspired by Flyway, and uses file-based SQL change scripts (versioned, repeatable, always).
• Key use cases: scanning a scripts folder tree, rendering and deploying scripts via Jinja templating, recording applied changes in a Snowflake change history table.
File Structure Analysis
• Top-level:
• schemachange/ → main package with modules:
– cli.py → entrypoint, command parsing (deploy, render).
– config/ → YAML & CLI args parsing, config merging, secrets filtering.
– session/ → Snowflake connection abstractions (SnowflakeSession, Script wrapper, Credential).
– JinjaTemplateProcessor.py & JinjaEnvVar.py → template engine integration.
– deploy.py → orchestrates the deploy command.
– redact_config_secrets.py → filters secrets in logs.
• demo/ → sample change-script projects (basics, citibike, Jinja demos).
• tests/ → pytest suite covering config, session, template processing, CLI misc, main.
• Dockerfile, workflows → CI/CD, packaging, publishing.
• Config files: pyproject.toml, setup.cfg, requirements.txt, .pre-commit, GitHub Actions YAMLs.
README Insights
• Explains architecture components: change scripts folder, Jinja support with modules folder, connections.toml for Snowflake credentials.
• Describes two main commands: deploy (scans root-folder, processes versioned/repeatable/always scripts, updates metadata table) and render (renders a single script).
• Shows DevOps flow: packaging as pip or Docker image, integrating with CI/CD.
a. Main Components
• CLI Entrypoint (schemachange/cli.py)
• Configuration Loader (YAML + CLI args + environment variables)
• Template Processor (Jinja engine + custom filters/env_var)
• ChangeScript Scanner & Parser (file system walker reading .sql/.sql.jinja scripts)
• Deployment Orchestrator (deploy.py): orders scripts (versioned → repeatable → always)
• SnowflakeSession Adapter: wraps Snowflake Python Connector
• ChangeHistory Table (Snowflake): METADATA.SCHEMACHANGE.CHANGE_HISTORY
• External Dependencies: Snowflake Python Connector, Jinja2, PyYAML, Pandas
• Demo Projects (on-disk scripts) and Modules Folder (optional Jinja macros)
b. Relationships & Data Flows
1. User invokes CLI (deploy or render) → cli.py parses commands.
2. Config Loader merges CLI args, schemachange-config.yml, connections.toml, env vars.
3. For deploy:
– File Scanner reads scripts under root-folder → yields ordered list.
– Template Processor applies Jinja templating (uses modules-folder for macros).
– Deployment Orchestrator invokes SnowflakeSession for each script:
· Connect to Snowflake → execute “USE” statements → run SQL.
· After execution, reset session context.
· Write metadata to ChangeHistory table.
4. For render:
– File Scanner locates specific script → Template Processor renders to console.
5. SnowflakeSession uses Snowflake Connector to reach Snowflake cloud service.
c. Architectural Patterns & Principles
• Layered architecture: CLI → Config → Processing → Persistence (Snowflake)
• Dependency Injection: passing connection, config, and template engine into orchestrator
• Single Responsibility: each module (config parser, template processor, session adapter) has a narrow focus
• Plugin-style extensibility: modules-folder for user-defined Jinja macros
• Idempotent, ordered migration pattern (versioned/repeatable/always)
d. Technologies & Frameworks
• Python 3.x
• Jinja2 for templating
• PyYAML for config
• Snowflake Python Connector (snowflake-connector-python)
• pytest for testing
• Docker for containerized runs
• GitHub Actions & tox for CI