# phdata/toolkit-snowflake-demo architecture

> Architecture diagram of the GitHub repository phdata/toolkit-snowflake-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phdata/toolkit-snowflake-demo
- Repository: https://github.com/phdata/toolkit-snowflake-demo

## Overview

##### Project Type and Purpose
- This is an open-source demo repository showcasing how to integrate the phData Toolkit CLI into a GitHub-based CI/CD pipeline targeting a Snowflake account.
- It contains multiple “tool” subprojects (access, advisor, data-generation, data-source) plus an infrastructure provisioning component.

##### File Structure Analysis
- Top-level directories:
- .github/ — GitHub Actions workflows and a reusable “common-setup” action
- access/, advisor/, data-generation/, data-source/ — each houses toolkit.conf and tool-specific code or artifacts
- provision/ — SQL scripts and CloudFormation/Jinja templates to spin up Snowflake resources
- Each tool directory follows the same pattern: a .gitignore, a toolkit.conf, plus subfolders for outputs (e.g., snapshots in data-source).
- Workflow triggers are defined under .github/workflows for scheduled/manual runs and for provisioning on PRs/merges.

##### README Insights
- Confirms that each tool directory corresponds to a separate CLI invocation.
- Explains that the “common-setup” action handles Java installation, Toolkit CLI install, and CLI authentication via secrets.

##### Components
a. Main Components
1. Developer/GitHub Actor
2. GitHub Repository
3. GitHub Actions Runner
4. Common-Setup Action (installs Java + Toolkit CLI + config)
5. phData Toolkit CLI (Java-based)
6. Tool Modules (access, advisor, data-generation, data-source)
7. Provisioning Engine (Jinja templates + CloudFormation/Snowflake SQL)
8. Snowflake Account (Database, Warehouses, Roles, Grants)
9. Local Workspace & Artifacts (JSON snapshots, specs)
10. GitHub Artifacts (optional – where JSON outputs might be uploaded)
b. Relationships & Data Flow
- Developer pushes code → GitHub repo → triggers Workflow
- Runner executes “common-setup” → Toolkit CLI ready
- For each tool (access, advisor, etc.):
- Workflow invokes Toolkit CLI with the tool’s toolkit.conf
- CLI connects to Snowflake via credentials from GitHub Secrets
- Performs actions (role setup, advisor checks, data gen, snapshots)
- Outputs JSON/artifacts into the tool’s directory or uploads as GitHub Artifact
- Provision workflow (on changes under provision/) invokes the Provisioning Engine → updates Snowflake infra
c. Architectural Patterns & Design Principles
- Modular “tool” pattern: each functional area isolated in its own directory
- Reusable CI step: common-setup action for DRY principle
- IaC provisioning via template-driven engine
- Secrets-based authentication and environment isolation per workflow
d. Key Technologies
- GitHub Actions (workflows, reusable actions)
- Java & phData Toolkit CLI
- Snowflake (cloud data warehouse)
- Jinja2 templates & CloudFormation/Snowflake SQL for provisioning

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CI/CD Layer
    subgraph "CI/CD Layer"
        Dev((Developer/GitHub Actor)):::ci
        GHRepo[(GitHub Repository)]:::ci
        Runner[(GitHub Actions Runner)]:::ci
        CommonSetup(["Common-Setup Action"]):::ci
    end

    %% Tooling Layer
    subgraph "Toolkit Execution"
        CLI(("phData Toolkit CLI")):::toolkit
        subgraph "Tool Modules"
            subgraph "Access Module"
                AConf["toolkit.conf"]:::tool
                AIgnore[".gitignore"]:::tool
            end
            subgraph "Advisor Module"
                BConf["toolkit.conf"]:::tool
                BIgnore[".gitignore"]:::tool
            end
            subgraph "Data-Generation Module"
                CConf["toolkit.conf"]:::tool
                CIgnore[".gitignore"]:::tool
                Spec["specs/advanced-snowflake-spec.yaml"]:::tool
            end
            subgraph "Data-Source Module"
                DConf["toolkit.conf"]:::tool
                DIgnore[".gitignore"]:::tool
                subgraph "Snapshots"
                    Diff["diff"]:::storage
                    Match["match"]:::storage
                    Profile["profile"]:::storage
                    Scan["scan"]:::storage
                end
            end
        end
    end

    %% Provisioning Layer
    subgraph "Provisioning (IaC)"
        ProvisionEngine(["Provisioning Engine"]):::iac
        PSQL["account-setup.sql"]:::iac
        PConf["toolkit.conf"]:::iac
        PTmpl["stack/templates/project.yaml.j2"]:::iac
        PRes["stack/resources"]:::iac
        PGroup["stack/groups/project.yaml"]:::iac
    end

    %% Artifacts
    subgraph "Artifacts & Storage"
        LocalArtifacts[(Local Workspace & Artifacts)]:::storage
        GHArtifacts[(GitHub Artifacts)]:::storage
    end

    %% External System
    subgraph "External System"
        Snowflake[(Snowflake Account)]:::external
    end

    %% Connections
    Dev -->|push code| GHRepo
    GHRepo -->|trigger workflows| Runner
    Runner -->|runs common steps| CommonSetup
    CommonSetup -->|sets up CLI| CLI

    CLI -->|runs tool| AConf
    CLI -->|runs tool| BConf
    CLI -->|runs tool| CConf
    CLI -->|runs tool| DConf

    AConf -->|reads config| Snowflake
    BConf -->|reads config| Snowflake
    CConf -->|reads config| Snowflake
    DConf -->|reads config| Snowflake

    CLI -->|writes outputs| LocalArtifacts
    CLI -->|uploads artifacts| GHArtifacts

    GHRepo -->|provision trigger| Runner
    Runner -->|executes IaC| ProvisionEngine
    ProvisionEngine -->|deploy infra| Snowflake

    %% Click Events
    click GHRepo "https://github.com/phdata/toolkit-snowflake-demo/tree/master//"
    click Runner "https://github.com/phdata/toolkit-snowflake-demo/blob/master/.github/workflows/access.yaml"
    click Runner "https://github.com/phdata/toolkit-snowflake-demo/blob/master/.github/workflows/advisor.yaml"
    click Runner "https://github.com/phdata/toolkit-snowflake-demo/blob/master/.github/workflows/data-generation.yaml"
    click Runner "https://github.com/phdata/toolkit-snowflake-demo/blob/master/.github/workflows/data-source.yaml"
    click Runner "https://github.com/phdata/toolkit-snowflake-demo/blob/master/.github/workflows/provision.yaml"
    click CommonSetup "https://github.com/phdata/toolkit-snowflake-demo/blob/master/.github/actions/common-setup/action.yaml"
    click AConf "https://github.com/phdata/toolkit-snowflake-demo/blob/master/access/toolkit.conf"
    click AIgnore "https://github.com/phdata/toolkit-snowflake-demo/blob/master/access/.gitignore"
    click BConf "https://github.com/phdata/toolkit-snowflake-demo/blob/master/advisor/toolkit.conf"
    click BIgnore "https://github.com/phdata/toolkit-snowflake-demo/blob/master/advisor/.gitignore"
    click CConf "https://github.com/phdata/toolkit-snowflake-demo/blob/master/data-generation/toolkit.conf"
    click CIgnore "https://github.com/phdata/toolkit-snowflake-demo/blob/master/data-generation/.gitignore"
    click Spec "https://github.com/phdata/toolkit-snowflake-demo/blob/master/data-generation/specs/advanced-snowflake-spec.yaml"
    click DConf "https://github.com/phdata/toolkit-snowflake-demo/blob/master/data-source/toolkit.conf"
    click DIgnore "https://github.com/phdata/toolkit-snowflake-demo/blob/master/data-source/.gitignore"
    click Diff "https://github.com/phdata/toolkit-snowflake-demo/tree/master/data-source/tools/ds/snapshots/diff"
    click Match "https://github.com/phdata/toolkit-snowflake-demo/tree/master/data-source/tools/ds/snapshots/match"
    click Profile "https://github.com/phdata/toolkit-snowflake-demo/tree/master/data-source/tools/ds/snapshots/profile"
    click Scan "https://github.com/phdata/toolkit-snowflake-demo/tree/master/data-source/tools/ds/snapshots/scan"
    click PSQL "https://github.com/phdata/toolkit-snowflake-demo/blob/master/provision/account-setup.sql"
    click PConf "https://github.com/phdata/toolkit-snowflake-demo/blob/master/provision/toolkit.conf"
    click PTmpl "https://github.com/phdata/toolkit-snowflake-demo/blob/master/provision/stack/templates/project.yaml.j2"
    click PRes "https://github.com/phdata/toolkit-snowflake-demo/tree/master/provision/stack/resources"
    click PGroup "https://github.com/phdata/toolkit-snowflake-demo/blob/master/provision/stack/groups/project.yaml"

    %% Styles
    classDef ci fill:#dff0d8,stroke:#3c763d,color:#3c763d;
    classDef toolkit fill:#d9edf7,stroke:#31708f,color:#31708f;
    classDef tool fill:#fcf8e3,stroke:#8a6d3b,color:#8a6d3b;
    classDef iac fill:#f2dede,stroke:#a94442,color:#a94442;
    classDef external fill:#f5f5f5,stroke:#333,color:#333;
    classDef storage fill:#eee8aa,stroke:#b5a642,color:#b5a642;
```
