How getsynq/dwhsupport works
Project Type and Purpose
• The project is a library (rather than a full-stack app) meant to unify access and simplify interactions with different data warehouse systems.
• It provides standardized interfaces, like the “Scrapper” interface, and a set of data models that represent table metrics, catalog details, SQL definitions, and more.
• The project is modular and extensible to support different targets (BigQuery, ClickHouse, Databricks, DuckDB, MySQL, Postgres, Redshift, Snowflake, standard SQL, etc.).
File Structure – Key Components
• Core Domain Logic:
– The “scrapper” directory contains multiple subdirectories for individual data warehouses. Each one has its own implementation of querying catalog information, table metrics, SQL definitions, etc. This reflects a plugin-like structure for handling different DWH systems.
– The “exec” directory includes similar segregation where different data warehouse connection and execution logic is separated by system (e.g., exec/bigquery, exec/clickhouse, etc.). This supports executing queries according to different DWH requirements.
• Interfaces and Models:
– The README mentions the “Scrapper” interface and various data models (TableMetricsRow, CatalogColumnRow, TableRow, SqlDefinitionRow, DatabaseRow) which encapsulate the DWH metadata.
– The “querybuilder” package handles dynamic SQL query construction, which is central to the library’s abstraction.
• Auxiliary Functionality:
– “logging” and “metrics” directories denote cross-cutting concerns: logging actions and capturing metrics for operations.
– The “connect” and “sshtunnel” directories deal with connections – including secure tunneling – which is essential for interacting with remote data warehouses.
– The “blocklist” and “lazy” packages may house configuration rules or deferred initialization logic used within the library.
• SQL Dialect Abstraction:
– “sqldialect” separates common SQL conditional logic into a distinct component, making it easier to tailor queries for each target DWH system.