# getsynq/dwhsupport architecture

> Architecture diagram of the GitHub repository getsynq/dwhsupport, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/getsynq/dwhsupport
- Repository: https://github.com/getsynq/dwhsupport

## Overview

##### Project Type and Purpose
- The project is a library (rather than a full-stack app) meant to unify access and simplify interactions with different data warehouse systems.
- It provides standardized interfaces, like the “Scrapper” interface, and a set of data models that represent table metrics, catalog details, SQL definitions, and more.
- The project is modular and extensible to support different targets (BigQuery, ClickHouse, Databricks, DuckDB, MySQL, Postgres, Redshift, Snowflake, standard SQL, etc.).

##### File Structure – Key Components
- Core Domain Logic:
– The “scrapper” directory contains multiple subdirectories for individual data warehouses. Each one has its own implementation of querying catalog information, table metrics, SQL definitions, etc. This reflects a plugin-like structure for handling different DWH systems.
– The “exec” directory includes similar segregation where different data warehouse connection and execution logic is separated by system (e.g., exec/bigquery, exec/clickhouse, etc.). This supports executing queries according to different DWH requirements.
- Interfaces and Models:
– The README mentions the “Scrapper” interface and various data models (TableMetricsRow, CatalogColumnRow, TableRow, SqlDefinitionRow, DatabaseRow) which encapsulate the DWH metadata.
– The “querybuilder” package handles dynamic SQL query construction, which is central to the library’s abstraction.
- Auxiliary Functionality:
– “logging” and “metrics” directories denote cross-cutting concerns: logging actions and capturing metrics for operations.
– The “connect” and “sshtunnel” directories deal with connections – including secure tunneling – which is essential for interacting with remote data warehouses.
– The “blocklist” and “lazy” packages may house configuration rules or deferred initialization logic used within the library.
- SQL Dialect Abstraction:
– “sqldialect” separates common SQL conditional logic into a distinct component, making it easier to tailor queries for each target DWH system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Level Client
    Client["Library Client"]:::client

    %% Core Domain
    subgraph "Core Domain"
        ScrapperInterface["Scrapper Interface"]:::core
        DataModels["Data Models"]:::core
    end

    Client -->|"calls"| ScrapperInterface

    %% Auxiliary Configuration
    subgraph "Auxiliary Configuration"
        Blocklist["Blocklist Functionality"]:::aux
        Lazy["Lazy Initialization"]:::aux
    end

    Blocklist -->|"configures"| ScrapperInterface
    Lazy -->|"initializes"| ScrapperInterface

    %% Data Warehouse Scrappers
    subgraph "Data Warehouse Scrappers"
        BigQueryScrapper["BigQuery Scrapper"]:::scrapper
        ClickHouseScrapper["ClickHouse Scrapper"]:::scrapper
        DatabricksScrapper["Databricks Scrapper"]:::scrapper
        DuckDBScrapper["DuckDB Scrapper"]:::scrapper
        MySQLScrapper["MySQL Scrapper"]:::scrapper
        PostgresScrapper["Postgres Scrapper"]:::scrapper
        RedshiftScrapper["Redshift Scrapper"]:::scrapper
        SnowflakeScrapper["Snowflake Scrapper"]:::scrapper
        StdSQLScrapper["Standard SQL Scrapper"]:::scrapper
    end

    %% From Scrapper Interface to each Scrapper Implementation
    ScrapperInterface -->|"implements"| BigQueryScrapper
    ScrapperInterface -->|"implements"| ClickHouseScrapper
    ScrapperInterface -->|"implements"| DatabricksScrapper
    ScrapperInterface -->|"implements"| DuckDBScrapper
    ScrapperInterface -->|"implements"| MySQLScrapper
    ScrapperInterface -->|"implements"| PostgresScrapper
    ScrapperInterface -->|"implements"| RedshiftScrapper
    ScrapperInterface -->|"implements"| SnowflakeScrapper
    ScrapperInterface -->|"implements"| StdSQLScrapper

    %% Query Generation Subgraph
    subgraph "Query Generation"
        QueryBuilder["Query Builder Module"]:::query
        SQLDialect["SQL Dialect Abstraction"]:::query
    end

    %% Dummy aggregator to represent query output
    QG["Query Generation Output"]:::queryDummy

    %% Flow within Query Generation
    ScrapperInterface -->|"buildsQuery"| QueryBuilder
    QueryBuilder -->|"appliesRules"| SQLDialect
    SQLDialect -->|"producesSQL"| QG

    %% Executor Modules Subgraph
    subgraph "Executor Modules"
        BigQueryExe["BigQuery Executor"]:::executor
        ClickHouseExe["ClickHouse Executor"]:::executor
        DatabricksExe["Databricks Executor"]:::executor
        DuckDBExe["DuckDB Executor"]:::executor
        MySQLExe["MySQL Executor"]:::executor
        PostgresExe["Postgres Executor"]:::executor
        RedshiftExe["Redshift Executor"]:::executor
        SnowflakeExe["Snowflake Executor"]:::executor
        StdSQLExe["Standard SQL Executor"]:::executor
    end

    %% Link Scrappers to their corresponding Executors
    BigQueryScrapper -->|"executes"| BigQueryExe
    ClickHouseScrapper -->|"executes"| ClickHouseExe
    DatabricksScrapper -->|"executes"| DatabricksExe
    DuckDBScrapper -->|"executes"| DuckDBExe
    MySQLScrapper -->|"executes"| MySQLExe
    PostgresScrapper -->|"executes"| PostgresExe
    RedshiftScrapper -->|"executes"| RedshiftExe
    SnowflakeScrapper -->|"executes"| SnowflakeExe
    StdSQLScrapper -->|"executes"| StdSQLExe

    %% Link Query Generation output to Executors
    QG -->|"executesSQL"| BigQueryExe
    QG -->|"executesSQL"| ClickHouseExe
    QG -->|"executesSQL"| DatabricksExe
    QG -->|"executesSQL"| DuckDBExe
    QG -->|"executesSQL"| MySQLExe
    QG -->|"executesSQL"| PostgresExe
    QG -->|"executesSQL"| RedshiftExe
    QG -->|"executesSQL"| SnowflakeExe
    QG -->|"executesSQL"| StdSQLExe

    %% Utilities Subgraph
    subgraph "Utilities"
        Logging["Logging Utility"]:::utility
        Metrics["Metrics Collection Module"]:::utility
        subgraph "Connection Management"
            CoreConnection["Core Connection"]:::connection
            SSHTunneling["SSH Tunneling"]:::connection
        end
    end

    %% From Executor Modules to Utilities
    BigQueryExe -->|"logsMetrics"| Logging
    ClickHouseExe -->|"logsMetrics"| Logging
    DatabricksExe -->|"logsMetrics"| Logging
    DuckDBExe -->|"logsMetrics"| Logging
    MySQLExe -->|"logsMetrics"| Logging
    PostgresExe -->|"logsMetrics"| Logging
    RedshiftExe -->|"logsMetrics"| Logging
    SnowflakeExe -->|"logsMetrics"| Logging
    StdSQLExe -->|"logsMetrics"| Logging

    BigQueryExe -->|"logsMetrics"| Metrics
    ClickHouseExe -->|"logsMetrics"| Metrics
    DatabricksExe -->|"logsMetrics"| Metrics
    DuckDBExe -->|"logsMetrics"| Metrics
    MySQLExe -->|"logsMetrics"| Metrics
    PostgresExe -->|"logsMetrics"| Metrics
    RedshiftExe -->|"logsMetrics"| Metrics
    SnowflakeExe -->|"logsMetrics"| Metrics
    StdSQLExe -->|"logsMetrics"| Metrics

    BigQueryExe -->|"usesConn"| CoreConnection
    ClickHouseExe -->|"usesConn"| CoreConnection
    DatabricksExe -->|"usesConn"| CoreConnection
    DuckDBExe -->|"usesConn"| CoreConnection
    MySQLExe -->|"usesConn"| CoreConnection
    PostgresExe -->|"usesConn"| CoreConnection
    RedshiftExe -->|"usesConn"| CoreConnection
    SnowflakeExe -->|"usesConn"| CoreConnection
    StdSQLExe -->|"usesConn"| CoreConnection

    CoreConnection -->|"securedBy"| SSHTunneling

    %% Styling
    classDef client fill:#ffdead,stroke:#000,stroke-width:2px;
    classDef core fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef scrapper fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef query fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef queryDummy fill:#e2e3e5,stroke:#6c757d,stroke-width:2px,stroke-dasharray: 5 5;
    classDef executor fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef utility fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef connection fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef aux fill:#e2f0cb,stroke:#6f42c1,stroke-width:2px;

    %% Click Events for Core Domain
    click ScrapperInterface "https://github.com/getsynq/dwhsupport/blob/main/scrapper/interface.go"
    click DataModels "https://github.com/getsynq/dwhsupport/blob/main/scrapper/models.go"

    %% Click Events for Data Warehouse Scrappers
    click BigQueryScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/bigquery/"
    click ClickHouseScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/clickhouse/"
    click DatabricksScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/databricks/"
    click DuckDBScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/duckdb/"
    click MySQLScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/mysql/"
    click PostgresScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/postgres/"
    click RedshiftScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/redshift/"
    click SnowflakeScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/snowflake/"
    click StdSQLScrapper "https://github.com/getsynq/dwhsupport/tree/main/scrapper/stdsql/"

    %% Click Events for Query Generation
    click QueryBuilder "https://github.com/getsynq/dwhsupport/blob/main/querybuilder/query_builder.go"
    click SQLDialect "https://github.com/getsynq/dwhsupport/blob/main/sqldialect/dialect.go"

    %% Click Events for Executor Modules
    click BigQueryExe "https://github.com/getsynq/dwhsupport/tree/main/exec/bigquery/"
    click ClickHouseExe "https://github.com/getsynq/dwhsupport/tree/main/exec/clickhouse/"
    click DatabricksExe "https://github.com/getsynq/dwhsupport/tree/main/exec/databricks/"
    click DuckDBExe "https://github.com/getsynq/dwhsupport/tree/main/exec/duckdb/"
    click MySQLExe "https://github.com/getsynq/dwhsupport/tree/main/exec/mysql/"
    click PostgresExe "https://github.com/getsynq/dwhsupport/tree/main/exec/postgres/"
    click RedshiftExe "https://github.com/getsynq/dwhsupport/tree/main/exec/redshift/"
    click SnowflakeExe "https://github.com/getsynq/dwhsupport/tree/main/exec/snowflake/"
    click StdSQLExe "https://github.com/getsynq/dwhsupport/tree/main/exec/stdsql/"

    %% Click Events for Utilities
    click Logging "https://github.com/getsynq/dwhsupport/blob/main/logging/logger.go"
    click Metrics "https://github.com/getsynq/dwhsupport/tree/main/metrics/"
    click CoreConnection "https://github.com/getsynq/dwhsupport/blob/main/connect/connect.go"
    click SSHTunneling "https://github.com/getsynq/dwhsupport/blob/main/sshtunnel/ssh_tunnel.go"
    click Blocklist "https://github.com/getsynq/dwhsupport/tree/main/blocklist/"
    click Lazy "https://github.com/getsynq/dwhsupport/tree/main/lazy/"
```
