# datafold/data-diff architecture

> Architecture diagram of the GitHub repository datafold/data-diff, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datafold/data-diff
- Repository: https://github.com/datafold/data-diff

## Overview

##### Main Components

a) Core Engine Layer:
- Main diff_tables.py, hashdiff_tables.py, and joindiff_tables.py components
- These represent the core comparison logic

b) Database Abstraction Layer:
- This below the core engine
- 'databases' directory with its multiple database connectors
- Connections to supported databases (PostgreSQL, MySQL, Snowflake, etc.)
- Use the abstract base classes (abcs/) as an interface between the core and database implementations

c) Query Generation Layer:
- This between the core engine and database layer
- 'queries' package with its components (api.py, base.py)
- How it translates diff operations into database-specific queries

d) Integration Layer:
- This at the top
- Dbt integration components (dbt.py, dbt_parser.py)
- Cloud integration capabilities (cloud/)

##### Component Relationships

- User input → Core Engine
- Core Engine → Query Generation Layer
- Query Generation → Database Abstraction Layer
- Database Abstraction Layer → Actual Databases
- Integration Layer → Core Engine

##### Important Elements to Highlight

a) Extensibility Points:
- Database connector interface
- Query generation system
- Integration points for dbt

b) Key Processes:
- Table difference computation
- Hash-based comparison
- Join-based comparison

##### Technical Details

- Support for multiple database types (shown in databases/)
- Query optimization and generation pipeline
- Error handling and validation flow
- Configuration management

Create clear sections for:
- Input Processing
- Core Comparison Logic
- Database Connectivity
- Query Generation
- External Integrations

Key Considerations:
- Modularity of the database connectors
- How the query generation system adapts to different databases
- Separation between core logic and database-specific implementations
- Configuration and validation flow

This architecture should be represented as a layered system with clear boundaries between components, showing how the tool maintains flexibility while supporting multiple database types and comparison strategies.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Integration
        DBT["DBT Integration"]:::integration
        DBTP["DBT Parser"]:::integration
        CLOUD["Cloud Integration"]:::integration
        CDS["Cloud Data Source"]:::integration
    end

    subgraph Core["Core Engine Layer"]
        DIFF["Main Diff Engine"]:::core
        HASH["Hash-based Comparison"]:::core
        JOIN["Join-based Comparison"]:::core
    end

    subgraph Query["Query Generation Layer"]
        QAPI["Query API"]:::query
        QBASE["Query Base"]:::query
        QUTIL["Query Utilities"]:::query
    end

    subgraph DBA["Database Abstraction Layer"]
        ABC["Abstract Base Classes"]:::db
        DCON["Database Connectors"]:::db
        DTYPE["Database Types Interface"]:::db
        DBASE["Base Database Implementation"]:::db
    end

    subgraph Support["Supporting Components"]
        CONFIG["Configuration Management"]:::support
        ERROR["Error Handling"]:::support
        SCHEMA["Schema Management"]:::support
    end

    subgraph Databases["External Databases"]
        PSQL["PostgreSQL"]:::external
        MYSQL["MySQL"]:::external
        SNOW["Snowflake"]:::external
        OTHER["Other DBs..."]:::external
    end

    %% Integration Layer Connections
    DBT --> DIFF
    DBTP --> DBT
    CLOUD --> DIFF
    CDS --> CLOUD

    %% Core Layer Connections
    DIFF --> HASH
    DIFF --> JOIN
    HASH --> QAPI
    JOIN --> QAPI

    %% Query Layer Connections
    QAPI --> QBASE
    QBASE --> QUTIL
    QUTIL --> ABC

    %% Database Abstraction Connections
    ABC --> DCON
    DTYPE --> DCON
    DBASE --> DCON
    DCON --> Databases

    %% Support Connections
    CONFIG --> DIFF
    ERROR --> DIFF
    SCHEMA --> DIFF

    %% Styling
    classDef core fill:#2374ab
    classDef db fill:#47a025
    classDef query fill:#d4b483
    classDef integration fill:#845ec2
    classDef support fill:#ff8066
    classDef external fill:#gray

    %% Click Events
    click DIFF "https://github.com/datafold/data-diff/blob/master/data_diff/diff_tables.py"
    click HASH "https://github.com/datafold/data-diff/blob/master/data_diff/hashdiff_tables.py"
    click JOIN "https://github.com/datafold/data-diff/blob/master/data_diff/joindiff_tables.py"
    click ABC "https://github.com/datafold/data-diff/tree/master/data_diff/abcs/"
    click DCON "https://github.com/datafold/data-diff/tree/master/data_diff/databases/"
    click DTYPE "https://github.com/datafold/data-diff/blob/master/data_diff/abcs/database_types.py"
    click DBASE "https://github.com/datafold/data-diff/blob/master/data_diff/databases/base.py"
    click QAPI "https://github.com/datafold/data-diff/blob/master/data_diff/queries/api.py"
    click QBASE "https://github.com/datafold/data-diff/blob/master/data_diff/queries/base.py"
    click QUTIL "https://github.com/datafold/data-diff/blob/master/data_diff/query_utils.py"
    click DBT "https://github.com/datafold/data-diff/blob/master/data_diff/dbt.py"
    click DBTP "https://github.com/datafold/data-diff/blob/master/data_diff/dbt_parser.py"
    click CLOUD "https://github.com/datafold/data-diff/tree/master/data_diff/cloud/"
    click CDS "https://github.com/datafold/data-diff/blob/master/data_diff/cloud/data_source.py"
    click CONFIG "https://github.com/datafold/data-diff/blob/master/data_diff/config.py"
    click ERROR "https://github.com/datafold/data-diff/blob/master/data_diff/errors.py"
    click SCHEMA "https://github.com/datafold/data-diff/blob/master/data_diff/schema.py"
```
