# erezsh/reladiff architecture

> Architecture diagram of the GitHub repository erezsh/reladiff, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/erezsh/reladiff
- Repository: https://github.com/erezsh/reladiff

## Overview

##### Project type and purpose
- Reladiff is an open-source, Python-based CLI tool and library for “diffing” large tables across heterogeneous databases.
- It provides both a command-line frontend and a programmable Python API, dispatching SQL workloads to various database engines (Postgres, Snowflake, Trino, etc.), computing hashes or joins in-DB, and returning/minimizing transferred data.

##### High-level analysis of the file structure
- Top level:
- Dockerfile & docker-compose.yml – define containers for development, testing, and optional Presto/Trino clusters.
- dev/ – scripts and Presto/Trino configs to bootstrap test/benchmark environs.
- reladiff/ – the core Python package:
– __main__.py → CLI entrypoint
– config.py → parses TOML/args, builds DiffJob
– databases/ → one connector module per supported database, all implementing a common interface (connect, execute, fetch hashes/rows)
– diff_tables.py, hashdiff_tables.py, joindiff_tables.py → diff-engine logic (divide-and-conquer, threaded, join-based)
– thread_utils.py → threadpool orchestration
– query_utils.py → SQL generation and parameterization
– utils.py, parse_time.py, info_tree.py → support utilities, metadata extraction
- tests/ – unit and integration tests covering CLI, API, diff logic, and connectors.
- docs/ – Sphinx docs, how-to guides, architectural write-ups.

##### README insights
- Emphasis on in-DB computation, threaded performance, JSON/“git-like” diffs, cross-DB vs same-DB modes.
- Supports >12 databases via pluggable driver modules.
- Docker/dev harness to spin up Presto or Trino for benchmarking.

##### Main components
a. User Interface
- CLI Frontend (reladiff/__main__.py)
- Python API (reladiff.connect_to_table, diff_tables)
b. Configuration Layer
- Argument/TOML parser (config.py) producing a DiffJob spec
c. Diff Orchestrator
- Engine Coordinator that selects Cross-DB vs Same-DB algorithm
- Thread Manager (thread_utils)
d. SQL Generation
- Query Builder (query_utils)
e. Database Connectors
- Generic DatabaseInterface
- Concrete modules: postgresql.py, snowflake.py, bigquery.py, trino.py, etc.
f. External Databases
- Supported relational & cloud warehouses
- Optionally local Presto/Trino clusters (via dev/docker-compose)
g. Result Aggregation & Output
- In-memory merging of diffs
- Formatters: JSON, text (+/–), materialize results back into a target DB

##### Relationships & interactions
1. User issues CLI command or invokes Python API
2. Config Layer parses inputs into a unified DiffJob
3. Orchestrator instantiates two TableClients via Database Connectors
4. Orchestrator chooses algorithm:
– Cross-DB: iterative hash queries → range splits → minimal row fetch
– Same-DB: single outer-join query (with optional materialization)
5. Orchestrator uses Thread Manager to parallelize hash and row fetch steps
6. SQL Generation creates the queries sent to each TableClient
7. TableClient issues SQL over DB driver → returns metadata/hashes/rows
8. Orchestrator aggregates partial results → passes to Output Formatter
9. Formatter emits JSON/git diff to STDOUT or writes a table

##### Architectural patterns & design principles
- Strategy Pattern: diff algorithm modules (hash-based vs join-based) plugged into the orchestrator
- Adapter Pattern: each database connector adapts a common interface to specific drivers and SQL dialects
- Divide-and-Conquer: cross-DB diff algorithm splits key ranges by hash segments
- Thread Pool for concurrency
- Single Responsibility: separation between parsing, orchestration, SQL building, DB connectivity, and formatting

##### Essential elements to capture
- Clear labels for each module (filename or role)
- CLI/API layer
- Core engine (orchestrator + algorithms)
- Adapters (connectors)
- External databases/services

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface Layer
    subgraph "User Interface"
        CLI["CLI Frontend"]:::core
        API["Python API"]:::core
    end

    %% Configuration Layer
    Config["Config Parser (config.py)"]:::core

    %% Core Orchestrator
    subgraph "Diff Engine"
        CoreCoord["Core Coordinator\n(diff_tables.py)"]:::core
        subgraph "Strategies"
            CrossDB["Cross-DB Strategy\n(hashdiff_tables.py)"]:::core
            SameDB["Same-DB (Join) Strategy\n(joindiff_tables.py)"]:::core
        end
        ThreadMgr["Thread Manager\n(thread_utils.py)"]:::core
        QueryBuilder["Query Builder\n(query_utils.py)"]:::core
    end

    %% Adapter Layer
    subgraph "Database Adapters"
        DBInterface["Database Interface\n(base.py)"]:::connector
        BigQuery["BigQuery Connector\n(bigquery.py)"]:::connector
        ClickHouse["ClickHouse Connector\n(clickhouse.py)"]:::connector
        Databricks["Databricks Connector\n(databricks.py)"]:::connector
        DuckDB["DuckDB Connector\n(duckdb.py)"]:::connector
        MySQL["MySQL Connector\n(mysql.py)"]:::connector
        Oracle["Oracle Connector\n(oracle.py)"]:::connector
        PostgreSQL["PostgreSQL Connector\n(postgresql.py)"]:::connector
        Presto["Presto Connector\n(presto.py)"]:::connector
        Redshift["Redshift Connector\n(redshift.py)"]:::connector
        Snowflake["Snowflake Connector\n(snowflake.py)"]:::connector
        Trino["Trino Connector\n(trino.py)"]:::connector
        Vertica["Vertica Connector\n(vertica.py)"]:::connector
        ConnectHelpers["Connector Helpers\n(_connect.py)"]:::connector
    end

    %% External Databases
    subgraph "External Databases" 
        direction TB
        PostgresExt["PostgreSQL"]:::external
        SnowflakeExt["Snowflake"]:::external
        TrinoExt["Trino"]:::external
        PrestoExt["Presto"]:::external
        OtherDBs["Other DBs (MySQL, Oracle, etc.)"]:::external
    end

    %% Output and Utilities
    subgraph "Result & Utilities"
        Utils["Metadata & Output Utils\n(utils.py)"]:::core
        InfoTree["Schema/Diff Inspector\n(info_tree.py)"]:::core
        TimeParse["Time Parser\n(parse_time.py)"]:::core
        Segments["Segment Representation\n(table_segment.py)"]:::core
        Formatter["Output Formatter"]:::output
    end

    %% Dev Harness
    subgraph "Development Harness" 
        DevHarness["Dev Scripts"]:::external
        DockerCompose["docker-compose.yml"]:::external
    end

    %% Connections
    CLI -->|invokes| Config
    API -->|calls| Config
    Config -->|builds DiffJob| CoreCoord
    CoreCoord -->|selects| CrossDB
    CoreCoord -->|selects| SameDB
    CoreCoord -->|uses| ThreadMgr
    CoreCoord -->|requests SQL| QueryBuilder
    QueryBuilder -->|generates| DBInterface
    DBInterface --> BigQuery
    DBInterface --> ClickHouse
    DBInterface --> Databricks
    DBInterface --> DuckDB
    DBInterface --> MySQL
    DBInterface --> Oracle
    DBInterface --> PostgreSQL
    DBInterface --> Presto
    DBInterface --> Redshift
    DBInterface --> Snowflake
    DBInterface --> Trino
    DBInterface --> Vertica
    BigQuery --> BigQueryExt["BigQuery"]:::external
    ClickHouse --> OtherDBs
    Databricks --> OtherDBs
    DuckDB --> OtherDBs
    MySQL --> OtherDBs
    Oracle --> OtherDBs
    PostgreSQL --> PostgresExt
    Presto --> PrestoExt
    Redshift --> OtherDBs
    Snowflake --> SnowflakeExt
    Trino --> TrinoExt
    Vertica --> OtherDBs
    ConnectHelpers --> DBInterface
    CrossDB -->|parallel hash queries| ThreadMgr
    SameDB -->|single join query| QueryBuilder
    ThreadMgr -->|executes tasks| DBInterface
    DBInterface -->|returns data| CoreCoord
    CoreCoord -->|aggregates| Utils
    Utils -->|formats| Formatter
    Formatter -->|outputs| CLI
    Formatter -->|outputs| API
    DevHarness -.-> PrestoExt
    DevHarness -.-> TrinoExt
    DockerCompose -.-> PrestoExt
    DockerCompose -.-> TrinoExt

    %% Click Events
    click CLI "https://github.com/erezsh/reladiff/blob/master/reladiff/__main__.py"
    click API "https://github.com/erezsh/reladiff/blob/master/reladiff/config.py"
    click Config "https://github.com/erezsh/reladiff/blob/master/reladiff/config.py"
    click CrossDB "https://github.com/erezsh/reladiff/blob/master/reladiff/hashdiff_tables.py"
    click SameDB "https://github.com/erezsh/reladiff/blob/master/reladiff/joindiff_tables.py"
    click CoreCoord "https://github.com/erezsh/reladiff/blob/master/reladiff/diff_tables.py"
    click ThreadMgr "https://github.com/erezsh/reladiff/blob/master/reladiff/thread_utils.py"
    click QueryBuilder "https://github.com/erezsh/reladiff/blob/master/reladiff/query_utils.py"
    click DBInterface "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/base.py"
    click BigQuery "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/bigquery.py"
    click ClickHouse "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/clickhouse.py"
    click Databricks "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/databricks.py"
    click DuckDB "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/duckdb.py"
    click MySQL "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/mysql.py"
    click Oracle "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/oracle.py"
    click PostgreSQL "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/postgresql.py"
    click Presto "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/presto.py"
    click Redshift "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/redshift.py"
    click Snowflake "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/snowflake.py"
    click Trino "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/trino.py"
    click Vertica "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/vertica.py"
    click ConnectHelpers "https://github.com/erezsh/reladiff/blob/master/reladiff/databases/_connect.py"
    click Utils "https://github.com/erezsh/reladiff/blob/master/reladiff/utils.py"
    click InfoTree "https://github.com/erezsh/reladiff/blob/master/reladiff/info_tree.py"
    click TimeParse "https://github.com/erezsh/reladiff/blob/master/reladiff/parse_time.py"
    click Segments "https://github.com/erezsh/reladiff/blob/master/reladiff/table_segment.py"
    click DevHarness "https://github.com/erezsh/reladiff/tree/master/dev/"
    click DockerCompose "https://github.com/erezsh/reladiff/blob/master/docker-compose.yml"

    %% Styles
    classDef core fill:#BBDEFB,stroke:#333,stroke-width:1px
    classDef connector fill:#C8E6C9,stroke:#333,stroke-width:1px
    classDef external fill:#CFD8DC,stroke:#333,stroke-width:1px
    classDef output fill:#FFF9C4,stroke:#333,stroke-width:1px
```
