# snowflake-labs/sc.ddlexportscripts architecture

> Architecture diagram of the GitHub repository snowflake-labs/sc.ddlexportscripts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/snowflake-labs/sc.ddlexportscripts
- Repository: https://github.com/snowflake-labs/sc.ddlexportscripts

## Overview

##### Project type and purpose
- This is neither a monolithic full-stack app nor a GUI tool; it is an extensible collection of command-line export scripts (“plugins”) targeted at different relational and Big Data engines (Teradata, SQL Server, Oracle, Redshift, Netezza, Vertica, DB2, Hive, BigQuery, Databricks).
- Its sole purpose is to extract DDL/code from various database platforms in formats ingestible by the SnowConvert migration engine.

##### File structure
- Top-level directories: one vendor per folder (e.g., Teradata/, SQLServer/, Oracle/, …). Each contains:
- executable scripts (shell, PowerShell, .bat) under bin/ or directly at root
- SQL templates or helper libraries under scripts/ or similar
- “AlternativeSQLServerExtractionMethods” holds community/PowerShell variants.
- Tests/ covers automated validation per vendor (e.g., Tests/Teradata/).
- CI pipeline definitions under .github/workflows drive code-quality, packaging, and release.

##### README for additional insights
- README confirms the pattern: “Export Scripts for X” linking each vendor’s subfolder.
- No central orchestration code—each module runs independently.

a. Main components
- Script Launcher / CLI (user entrypoint)
- Vendor-Specific Export Modules (one per platform)
- Output: DDL Files (flat files or STDOUT)
- SnowConvert Migration Engine (external service/consumer)
- Test Harness (internal QA)
- CI/CD Pipelines (.github workflows)
b. Relationships & interactions
1. User/Automation triggers CLI
2. CLI invokes chosen Vendor Module
3. Module connects to target DB instance → extracts DDL → writes to storage (local FS or stream)
4. Extracted artifacts passed to SnowConvert (via API or file drop)
5. Test Harness invokes modules in containerized DB environments (e.g., Docker) and verifies expected outputs
6. CI/CD workflows run linting, tests, packaging, and release asset uploads
c. Architectural patterns
- Plugin-style modularity: each vendor’s scripts are isolated and interchangeable
- Separation of concerns: extraction logic vs. testing vs. CI/CD
- Script-based orchestration (no heavy frameworks)
d. Key technologies
- Shell scripts (bash, .sh)
- PowerShell scripts (.ps1)
- Python modules (Vertica, possibly Databricks)
- CI: GitHub Actions
- Containerization for Tests: Docker (e.g., DB2, Teradata)

##### Capturing essential architectural elements
- Shared utilities or configuration files (e.g., .env) if relevant.

## Diagram (Mermaid)

```mermaid
flowchart TD
    CLI["Script Launcher / CLI"]:::orchestration

    subgraph "Vendor Extraction Modules"
        Teradata["Teradata Module (bash)"]:::extraction
        Oracle["Oracle Module (bash)"]:::extraction
        Redshift["Redshift Module (bash)"]:::extraction
        BigQuery["BigQuery Module (bash)"]:::extraction
        DB2Mod["DB2 Module (bash)"]:::extraction
        Hive["Hive Module (bash)"]:::extraction
        Vertica["Vertica Module (python)"]:::extraction
        Databricks["Databricks Module (.dbc)"]:::extraction
        AltSQLServer["Alt SQLServer Module (PowerShell)"]:::extraction

        TeradataTemp["Teradata SQL Templates"]:::extraction
        OracleScripts["Oracle SQL Scripts"]:::extraction
        RedshiftScripts["Redshift SQL Scripts"]:::extraction
        VerticaUtils["Vertica Conversion Utilities"]:::extraction
    end

    subgraph "Relational & Big Data Engines"
        TeradataDB["Teradata DB"]:::external
        OracleDB["Oracle DB"]:::external
        RedshiftDB["Redshift"]:::external
        BigQueryDB["BigQuery"]:::external
        DB2DB["DB2"]:::external
        HiveDB["Hive"]:::external
        VerticaDB["Vertica"]:::external
        DatabricksDB["Databricks"]:::external
        SQLServerDB["SQL Server"]:::external
    end

    DDL["DDL Output (files/STDOUT)"]:::external
    SnowConv["SnowConvert Migration Engine"]:::external

    TestHarness["Test Harness (Containerized)"]:::test
    TeradataTests["Teradata Tests"]:::test
    DB2Tests["DB2 Tests"]:::test

    subgraph "CI/CD Pipelines"
        CodeQL["CodeQL Analysis"]:::orchestration
        Package["Package Scripts"]:::orchestration
        Release["Upload Release Assets"]:::orchestration
    end

    CLI --> Teradata
    CLI --> Oracle
    CLI --> Redshift
    CLI --> BigQuery
    CLI --> DB2Mod
    CLI --> Hive
    CLI --> Vertica
    CLI --> Databricks
    CLI --> AltSQLServer

    Teradata --> TeradataDB
    Oracle --> OracleDB
    Redshift --> RedshiftDB
    BigQuery --> BigQueryDB
    DB2Mod --> DB2DB
    Hive --> HiveDB
    Vertica --> VerticaDB
    Databricks --> DatabricksDB
    AltSQLServer --> SQLServerDB

    Teradata --> TeradataTemp
    Oracle --> OracleScripts
    Redshift --> RedshiftScripts
    Vertica --> VerticaUtils

    Teradata --> DDL
    Oracle --> DDL
    Redshift --> DDL
    BigQuery --> DDL
    DB2Mod --> DDL
    Hive --> DDL
    Vertica --> DDL
    Databricks --> DDL
    AltSQLServer --> DDL

    DDL --> SnowConv

    TestHarness --> Teradata
    TestHarness --> DB2Mod
    TestHarness --> TeradataDB
    TestHarness --> DB2DB
    TestHarness --> TeradataTests
    TestHarness --> DB2Tests

    CodeQL --> CLI
    CodeQL --> Teradata
    CodeQL --> Oracle
    CodeQL --> Redshift
    CodeQL --> DB2Mod

    Package --> CLI
    Package --> Teradata
    Package --> Oracle
    Package --> Redshift
    Package --> DB2Mod

    Release --> Package

    classDef extraction fill:#ADD8E6,stroke:#0000FF
    classDef orchestration fill:#90EE90,stroke:#008000
    classDef external fill:#FFA500,stroke:#FF8C00
    classDef test fill:#DDA0DD,stroke:#800080

    click Teradata "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Teradata/bin/create_ddls.sh"
    click Oracle "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Oracle/bin/create_ddls.sh"
    click Redshift "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Redshift/bin/create_ddls.sh"
    click BigQuery "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/BigQuery/bin/create_ddls.sh"
    click DB2Mod "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/DB2/bin/create_ddls.sh"
    click Hive "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Hive/exp_ddl.sh"
    click Vertica "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Vertica/Scripts/vertMain.py"
    click Databricks "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Databricks/Download_jobs_sources.dbc"
    click AltSQLServer "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/AlternativeSQLServerExtractionMethods/PowerShellScripts/bin/extract-sql-server-ddl.ps1"
    click TeradataTemp "https://github.com/snowflake-labs/sc.ddlexportscripts/tree/main/Teradata/scripts_template/"
    click OracleScripts "https://github.com/snowflake-labs/sc.ddlexportscripts/tree/main/Oracle/scripts/"
    click RedshiftScripts "https://github.com/snowflake-labs/sc.ddlexportscripts/tree/main/Redshift/scripts/"
    click VerticaUtils "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Vertica/Scripts/SFConfig.py"
    click VerticaUtils "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Vertica/Scripts/SFConvert.py"
    click VerticaUtils "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Vertica/Scripts/VerticaConfig.py"
    click VerticaUtils "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/Vertica/Scripts/VerticaDBCalls.py"
    click VerticaUtils "https://github.com/snowflake-labs/sc.ddlexportscripts/tree/main/Vertica/Scripts/SQL_Convert/"
    click TeradataTests "https://github.com/snowflake-labs/sc.ddlexportscripts/tree/main/Tests/Teradata/"
    click DB2Tests "https://github.com/snowflake-labs/sc.ddlexportscripts/tree/main/Tests/DB2/"
    click CodeQL "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/.github/workflows/codeql-analysis.yml"
    click Package "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/.github/workflows/package-scripts.yml"
    click Release "https://github.com/snowflake-labs/sc.ddlexportscripts/blob/main/.github/workflows/upload-release-assets.yml"
```
