How erezsh/reladiff works
Project type and purpose
• Reladiff is an open-source, Python-based CLI tool and library for “diffing” large tables across heterogeneous databases.
• It provides both a command-line frontend and a programmable Python API, dispatching SQL workloads to various database engines (Postgres, Snowflake, Trino, etc.), computing hashes or joins in-DB, and returning/minimizing transferred data.
High-level analysis of the file structure
• Top level:
• Dockerfile & docker-compose.yml – define containers for development, testing, and optional Presto/Trino clusters.
• dev/ – scripts and Presto/Trino configs to bootstrap test/benchmark environs.
• reladiff/ – the core Python package:
– __main__.py → CLI entrypoint
– config.py → parses TOML/args, builds DiffJob
– databases/ → one connector module per supported database, all implementing a common interface (connect, execute, fetch hashes/rows)
– diff_tables.py, hashdiff_tables.py, joindiff_tables.py → diff-engine logic (divide-and-conquer, threaded, join-based)
– thread_utils.py → threadpool orchestration
– query_utils.py → SQL generation and parameterization
– utils.py, parse_time.py, info_tree.py → support utilities, metadata extraction
• tests/ – unit and integration tests covering CLI, API, diff logic, and connectors.
• docs/ – Sphinx docs, how-to guides, architectural write-ups.
README insights
• Emphasis on in-DB computation, threaded performance, JSON/“git-like” diffs, cross-DB vs same-DB modes.
• Supports >12 databases via pluggable driver modules.
• Docker/dev harness to spin up Presto or Trino for benchmarking.
Main components
a. User Interface
• CLI Frontend (reladiff/__main__.py)
• Python API (reladiff.connect_to_table, diff_tables)
b. Configuration Layer
• Argument/TOML parser (config.py) producing a DiffJob spec
c. Diff Orchestrator
• Engine Coordinator that selects Cross-DB vs Same-DB algorithm
• Thread Manager (thread_utils)
d. SQL Generation
• Query Builder (query_utils)
e. Database Connectors
• Generic DatabaseInterface
• Concrete modules: postgresql.py, snowflake.py, bigquery.py, trino.py, etc.
f. External Databases
• Supported relational & cloud warehouses
• Optionally local Presto/Trino clusters (via dev/docker-compose)
g. Result Aggregation & Output
• In-memory merging of diffs
• Formatters: JSON, text (+/–), materialize results back into a target DB
Relationships & interactions
1. User issues CLI command or invokes Python API
2. Config Layer parses inputs into a unified DiffJob
3. Orchestrator instantiates two TableClients via Database Connectors
4. Orchestrator chooses algorithm:
– Cross-DB: iterative hash queries → range splits → minimal row fetch
– Same-DB: single outer-join query (with optional materialization)
5. Orchestrator uses Thread Manager to parallelize hash and row fetch steps
6. SQL Generation creates the queries sent to each TableClient
7. TableClient issues SQL over DB driver → returns metadata/hashes/rows
8. Orchestrator aggregates partial results → passes to Output Formatter
9. Formatter emits JSON/git diff to STDOUT or writes a table
Architectural patterns & design principles
• Strategy Pattern: diff algorithm modules (hash-based vs join-based) plugged into the orchestrator
• Adapter Pattern: each database connector adapts a common interface to specific drivers and SQL dialects
• Divide-and-Conquer: cross-DB diff algorithm splits key ranges by hash segments
• Thread Pool for concurrency
• Single Responsibility: separation between parsing, orchestration, SQL building, DB connectivity, and formatting
Essential elements to capture
• Clear labels for each module (filename or role)
• CLI/API layer
• Core engine (orchestrator + algorithms)
• Adapters (connectors)
• External databases/services